Skip to main content

This week at Dozer #11

ยท 5 min read
Abhishek

๐ŸŽ‰ Welcome to the New Release v0.1.24 of Dozer!

We are excited to announce the latest release of Dozer - v0.1.24. Our team has been working hard to enhance and refine the functionality of Dozer, and we are eager for you to experience all the improvements and new features. Let's delve into the details.

๐Ÿš€ Release v0.1.24 Highlights ๐Ÿš€โ€‹

Here are the exciting new features and improvements in our latest release, v0.1.24!

  • Show status of all API servers (#1601): This feature allows users to view the status of all API servers, providing a comprehensive overview of the system's health.
  • Emit metrics of source operations and pipeline latency (#1610): This feature is a significant enhancement to Dozer's monitoring capabilities. This feature allows Dozer to emit metrics related to source operations and pipeline latency, providing valuable insights into system performance.

    The metrics endpoint now includes the number of inserts from the source, represented by the source_insert counter. This counter provides a clear view of the data flow from the source into Dozer, helping users understand the rate of data ingestion.

    Additionally, the feature introduces the pipeline_latency summary, which measures the pipeline processing latency in seconds. This metric is crucial for understanding the efficiency of the data processing pipeline and identifying potential bottlenecks or performance issues.

    The pipeline_latency summary includes several quantiles (0, 0.5, 0.9, 0.95, 0.99, 0.999, and 1), each providing a different perspective on the latency distribution. For example, the 0.99 quantile represents the latency under which 99% of the observations fall.

  • Check for running environment (#1608):This feature allows Dozer to check its running environment, which can be set using the DOZER_ENV environment variable. This environment variable can be set up on the cluster, and it is not typically set during local runs. When running on a cluster, it will provide the appropriate value, while during local runs, it will default to "local".

  • Upgrade rustyline to 11.0.0 (#1613): We have upgraded rustyline to version 11.0.0, improving the overall performance and stability of Dozer.

  • Add token param in case access config endpoint by REST URL at server 8082 (#1615): feature is a significant enhancement to the security and flexibility of Dozer's configuration endpoint. This update introduces a new config_token parameter to the Dozer CLI. This token is attached as a bearer token if the configuration URL uses the HTTP/HTTPS protocol. This process ensures that only authorized entities can access the configuration endpoint, enhancing the security of the system.

  • Emit special error message for MapFull (#1618): This feature allows Dozer to emit a special error message when the Map is Full. The MapFull error typically occurs when the memory allocation for a cache is full, and no more items can be added.

  • Emit Product processor metrics (#1611): This feature is a significant addition to Dozer's monitoring capabilities. This feature allows Dozer to emit metrics related to the Product processor on the Prometheus endpoint. These metrics provide valuable insights into the performance and efficiency of the Product processor.

    The metrics include:

    • product_unsatisfied_joins: This counter represents the number of operations that did not match the Join condition, providing insights into the data that did not meet the specified criteria.

    • product_output_operations and product_input_operations: These counters represent the number of output and input operations, respectively, providing a clear view of the data flow in the Product processor.

    • product_right_lookup_size and product_left_lookup_size: These gauges represent the total number of items in the right and left lookup tables, respectively. These metrics provide insights into the size and complexity of the join operations.

    • product_latency: This summary measures the processing latency in the Product processor. It includes several quantiles (0, 0.5, 0.9, 0.95, 0.99, 0.999, and 1), each providing a different perspective on the latency distribution. This metric is crucial for understanding the efficiency of the Product processor and identifying potential performance issues.

  • Log client didn't authenticate itself (#1620): We have resolved an issue where the client did not authenticate itself, enhancing the security and reliability of the system.

We are immensely proud of the progress we have made with the latest v0.1.24 release. Our team continues to work relentlessly to enhance Dozer, and we deeply appreciate the feedback and contributions from our incredible community. Let's continue innovating together! ๐Ÿš€

Dozer v0.1.24 is available now. Check out the release notes here.

Looking Forward ๐ŸŒˆโ€‹

As we carry on with our efforts to improve Dozer, we are dedicated to the augmentation of its features and capabilities. Your feedback plays a pivotal role in our development process, so please don't hesitate to share your thoughts and ideas. You can always open an issue for feature request OR start a discussion thread under our GitHub Q&A category. Together, we can make Dozer the best it can be! โœจ ๐Ÿง™โ€โ™€๏ธ

Full Changelog:โ€‹

Contact us ๐Ÿ“ฌโ€‹