Skip to main content

This week at Dozer #12

Β· 5 min read
Abhishek

πŸŽ‰ Welcome to the New Release v0.1.25 of Dozer!

We are thrilled to present you with the latest release of Dozer - v0.1.25. Our team has diligently been working to enhance and refine the functionality of Dozer, and we are eager for you to experience all the improvements and new features. Let's delve into the details.

πŸš€ Release v0.1.25 Highlights πŸš€β€‹

Here are the exciting new features and improvements in our latest release, v0.1.25!

  • Introducing Kafka Connector in Dozer: Dozer now provides a way to ingest data from Kafka, a distributed streaming platform. Several improvements and updates have been made to the Kafka Connector, including a refactor to use a schema registry, a switch to rdkafka, the ability to use Kafka without a schema registry, and the addition of Kafka to the Docker image. Here's a sample configuration for the Kafka Connector:

    app_name: dozer-kafka
    connections:
    - config : !Kafka
    broker: {{broker_url}} # e.g. kafka-broker:9092
    schema_registry_url: {{schema_registry_url}} # e.g. http://localhost:8081
    name: kafka_store
    sources:
    - name: purchases
    table_name: purchases # kafka topic name
    connection: !Ref kafka_store
    columns:

    Please note that this feature is currently under a feature flag and needs to be explicitly enabled. To use the Kafka connector, you need to build Dozer from the source with the Kafka feature enabled.

    Use the following command to do so: cargo build --release --features kafka --bin dozer

    For more information, check out the Kafka Connector documentation and the Kafka Connector sample.

  • Dozer Now Supports Array of Character-Based Data Types (#1645): Dozer has now expanded its support to include array of character-based data types. This includes types such as text[], varchar[], char[], and bpchar[]. Internally, these data types are handled using JsonValue::Array, enhancing the flexibility and versatility of data processing within Dozer.

  • Add more cache metrics (#1627): This PR adds more cache metrics, providing deeper insights into cache performance and usage.

      # HELP cache_update Number of updates processed by cache builder
    # TYPE cache_update counter
    cache_update{endpoint="trips_cache",migration="v0053"} 18477

    # HELP build_index Number of operations built into indexes
    # TYPE build_index counter
    build_index{endpoint="trips_cache",migration="v0053",secondary_index="0",secondary_index_type="SortedInverted"} 36979
    build_index{endpoint="trips_cache",migration="v0053",secondary_index="1",secondary_index_type="SortedInverted"} 36979
    build_index{endpoint="trips_cache",migration="v0053",secondary_index="4",secondary_index_type="SortedInverted"} 36889
    build_index{endpoint="trips_cache",migration="v0053",secondary_index="3",secondary_index_type="SortedInverted"} 36979
    build_index{endpoint="trips_cache",migration="v0053",secondary_index="2",secondary_index_type="SortedInverted"} 36979

    # HELP cache_operation Number of operations stored in the cache
    # TYPE cache_operation counter
    cache_operation{endpoint="trips_cache",migration="v0053"} 36979

    # HELP cache_insert Number of inserts processed by cache builder
    # TYPE cache_insert counter
    cache_insert{endpoint="trips_cache",migration="v0053"} 25
  • Error Threshold Management (#1555): This feature enhances the resilience of the Dozer pipeline, preventing it from being abruptly terminated due to every single internal error it encounters.

    For instance, when running dozer -- -e 10, the pipeline will continue to operate despite encountering an SQL error, such as a division operation that cannot be executed. The system will log the error and continue processing, as shown in the logs below:

      INFO Starting gRPC server on http://0.0.0.0:50051 with security: None
    INFO starting 8 workers
    INFO Tokio runtime found; starting in existing Tokio runtime
    INFO [pg] Starting replication
    ERROR Sql: SQL Error: Division operation cannot be done.
    β–Ήβ–Ήβ–Ήβ–Ήβ–Ή err_threshold_test: 0: 0/s
    β–Έβ–Ήβ–Ήβ–Ήβ–Ή res: 1: 27.9485/s
    β–Ήβ–Ήβ–Ήβ–Ήβ–Ή err_threshold_test: 0: 0/s
    β–Ήβ–Ήβ–Ήβ–Ήβ–Ή res: 12: 330.1196/s
    β–Ήβ–Ήβ–Ήβ–Ήβ–Ή err_threshold_test: 0: 0/s

    This feature significantly improves the robustness of the Dozer pipeline, allowing it to handle errors gracefully and continue processing data.

  • Implement cache dump and restore (#1637): Introduces three key functions to facilitate cache dump and restore:

    begin_dump_txn: This function initiates a DumpTransaction, which is designed to dump the entire cache.

    dump: This function acts as a generator, producing zero-copy data slices that comprise the complete cache dump.

    restore: This function enables the restoration of the cache from an AsyncRead.

    In addition to these functions, a command-line utility named dozer-cache has been added. This utility can be used to count the number of records in a cache, dump a cache to a file, or restore a cache from a previously created dump file.

  • Remove maturin sccache because it keeps failing (#1623): This PR removes the maturin sccache due to consistent failures, improving the stability of the build process.

  • Add api latency metric (#1642): This PR adds an API latency metric, providing valuable insights into API performance.

We are immensely proud of the progress we have made with the latest v0.1.25 release. Our team continues to work relentlessly to enhance Dozer, and we deeply appreciate the feedback and contributions from our incredible community. Let's continue innovating together! πŸš€

Dozer v0.1.25 is available now. Check out the release notes here.

Looking Forward πŸŒˆβ€‹

As we carry on with our efforts to improve Dozer, we are dedicated to the augmentation of its features and capabilities. Your feedback plays a pivotal role in our development process, so please don't hesitate to share your thoughts and ideas. You can always open an issue for feature request OR start a discussion thread under our GitHub Q&A category. Together, we can make Dozer the best it can be! ✨ πŸ§™β€β™€οΈ

Full Changelog:​

Contact us πŸ“¬β€‹