Skip to main content

This week at Dozer #14

· 14 min read
Cahyo Subroto

🎉 Welcome to the New Release v0.1.29 of Dozer! 🎉

We're thrilled to announce that our dedicated team has invested immense effort into elevating and fine-tuning the capabilities of Dozer with the introduction of the latest release-v0.1.29.This latest version brings numerous bug fixes, enhancements, and a range of exciting new features. We're truly excited for you to delve into the remarkable enhancements we've implemented. Let's take a closer look at the specifics of this release:

🚀Release v0.1.29 Highlights🚀

Get ready to experience the exhilaration of our latest release as we unveil an array of thrilling new features and improvements. We can't wait to showcase what's in store for you!

  • Add "list deployments" endpoint (#1724): In this chore, a new endpoint has been added to the API that allows users to list all of the deployments that have been created. This endpoint is useful for administrators who need to track the status of all of the deployments in their system. The new endpoint is located at /api/v1/deployments. The JSON object has the following fields:
    • id: The unique identifier of the deployment.
    • name: The name of the deployment.
    • status: The status of the deployment (e.g., "running", "failed", "succeeded").
    • created_at: The date and time the deployment was created.
    • updated_at: The date and time the deployment was last updated.
  • Set bucket for histogram (#1723): This chore updates the code to set the bucket size for the histogram. The previous code used a fixed bucket size of 100. This meant that the histogram would not be evenly spaced if there were a small number of data points. For example, if there were only 5 data points, the histogram would have 5 buckets, each of which would contain 1 data point. The new code uses a dynamic bucket size. The bucket size is calculated based on the number of data points. This ensures that the histogram is always evenly spaced, regardless of the number of data points. For additional details, please refer to the bucket for histogram documentation.

  • Support reading of configs from multiple files (#1718): This feat added support for reading configuration files from multiple files. This change allows users to store their configuration data in multiple files, which can make it easier to manage and maintain their configuration. The previous code only allowed users to read configuration data from a single file. This meant that if a user had a large amount of configuration data, they would have to store it all in a single file. This could make the configuration file difficult to manage and maintain. The new code allows users to store their configuration data in multiple files. This makes it easier to manage and maintain the configuration data. For example, a user could store their database configuration in one file, their application configuration in another file, and their environment variables in a third file. With these changes, we support reading from multiple files. By default, it reads based on two patterns: dozer-config.* andqueries/*.sql. Additionally, users can provide patterns with the -c parameter, e.g. -c ./dozer-config.yaml -c ./dozer-config.sources.yaml -c ./queries/simple.sql. For additional details, please refer to the reading of configs from multiple files documentation.

  • Revert set bucket for histogram (#1732): In this chore, the changes made in pull request #1723 were reverted The changes set the bucket size for the histogram. However, this change caused some problems with the histogram. One problem was that the histogram was not always evenly spaced. This was because the bucket size was calculated based on the number of data points. If there were a small number of data points, the bucket size would be too large. This would cause the histogram to be unevenly spaced. Another problem was that the histogram was not always accurate. This was because the bucket size was not always large enough to contain all of the data points. This would cause some data points to be outside of the histogram. For more information, check the revert set bucket for histogram documentation.

  • Rename num replicas to num api instances (#1725): This chore renamed the num_replicas configuration option to num_api_instances. This change was made to make the configuration option more descriptive. The previous name, num_replicas, was ambiguous. It could be interpreted as the number of replicas of the entire application or the number of replicas of the API server. The new name, num_api_instances, makes it clear that the configuration option refers to the number of replicas of the API server. For more detailed information, please refer to the rename num replicas to num api instances.

  • Show deployment logs from deploy API (#1731): This feat added the ability to show deployment logs from the deploy API. This change allows users to view the logs for deployment without having to access the underlying infrastructure. The previous API only returned the status of the deployment. This meant that users had to access the underlying infrastructure to view the logs. This could be difficult and time-consuming. The new API returns the status of the deployment and the logs for the deployment. This makes it easier for users to view the logs for deployment without having to access the underlying infrastructure. This change is a valuable addition to the API and will make it easier for users to troubleshoot deployments.For details on how the show deployment logs from deploy API works click here.

  • Edit Kafka feature with Snowflake Docker image release flow: (**#1730) This chore edited the Kafka feature to use the Snowflake Docker image release flow. This change was made to make the Kafka feature more reliable and easier to maintain. The previous Kafka feature used a custom Docker image. This image was built and maintained by the Dozer team. This made it difficult to update the Kafka image when there were new security patches or bug fixes. The new Kafka feature uses the Snowflake Docker image release flow. This flow uses a pre-built Kafka image that is released by Snowflake. This makes it easier to update the Kafka image when there are new security patches or bug fixes. Here's a sample configuration for the Edit Kafka feature with Snowflake Docker image release flow:

 ${{ runner.os }}-cargo-release-${{ hashFiles('Cargo.lock') }}
${{ runner.os }}-cargo-release-
- name: Cargo build
uses: actions-rs/cargo@v1
with:
@@ -350,13 +351,9 @@ jobs:
if: matrix.os != 'ubuntu-20-16-cores'
run: cargo build--release --bin ${{ matrix.file\_name }}

- name: Buildpackagefor ubuntu (with kafka)
- name: Buildpackagefor ubuntu (with kafka & snowflake)
if: matrix.os == 'ubuntu-20-16-cores'
run: cargo build--release --bin ${{ matrix.file\_name }} --features=kafka

# - name: Build Linux binary
# if: matrix.os == 'ubuntu-20-16-cores'
# run: cargo build--features=snowflake --release
run: cargo build--release --bin ${{ matrix.file\_name }} --features "kafka snowflake"

- name: Preparerelease assets
shell: bash
  • Show more information when configuration not found (#1734): This pull refactored the code to show more information when a configuration is not found. This change makes it easier for users to troubleshoot configuration errors. The previous code would simply return an error message if a configuration was not found. This message was not very helpful for users who were trying to troubleshoot the error. For further details related to "Show more information when configuration not found", please click here.
  • Implement config override (#1736): This feat implemented the ability to override configuration values. This change allows users to customize the behavior of Dozer without having to modify the code. The previous version of Dozer used a single configuration file. This meant that all users had to use the same configuration values. This could be inconvenient for users who wanted to customize the behavior of Dozer. The new version of Dozer allows users to override configuration values with a new repeated command line parameter: --config-overrides. This parameter allows users to override configuration values from the command line. Example: dozer --config-overrides /app/commit_size=100. The part before = is a JSON pointer, and the part after = should be a valid JSON string. If the JSON pointer points to an existing config value, the JSON value will be replaced.
  • MAX_VALUE(expr1, expr2) (#1733): This feat added a new function called MAX_VALUE(expr1, expr2). This function returns the maximum value of two expressions. The new function makes it easier for users to calculate the maximum value of two expressions. The function is also more efficient than writing the code to calculate the maximum value. Here is an example of how to use the MAX_VALUE function:
defmax\_value(expr1, expr2):
"""Calculates the maximum value of two expressions."""
return max(expr1, expr2)

print(max\_value(10, 20))
# 20

note

Check out the documentation for further details.

  • Remove outdated file (#1741): This chore involves the removal of an outdated file from the Dozer codebase. This file was a copy of the Dozer documentation that was outdated. The outdated file was no longer needed and was taking up space in the codebase. Removing the file made the codebase smaller and easier to maintain. For a more in-depth understanding, we recommend exploring our extensive documents.
  • Update command descriptions (#1739): The chore updated the descriptions of the commands in the Dozer documentation. This change made the documentation more accurate and informative. The previous descriptions were outdated and incomplete. This made it difficult for users to understand what the commands did.To gain further insights into the command descriptions, consider reviewing the documents.
  • --config-overrides was not exposed in CLI (#1743): This edit fixed a bug where the --config-overrides command line parameter was not exposed in the CLI. This meant that users could not use the parameter to override configuration values from the command line. The bug was caused by a typo in the code. The typo was fixed and the --config-overrides parameter is now exposed in the CLI.This fix is a valuable improvement to the CLI. It makes it easier for users to override configuration values from the command line. Here is a sample configuration of how the--config-overrides was not exposed in CLI:
pub config\_paths: Vec\<String\>,
#[arg(global = true, long, hide = true)]
pub config\_token: Option\<String\>,
#[arg(long, value\_parser(parse\_config\_override))]
#[arg(global = true, long, value\_parser(parse\_config\_override))]
pub config\_overrides: Vec\<(String, serde\_json::Value)\>,

#[clap(subcommand)]
  • Serve replication log on internal grpc server (#1722): This feat added the ability to serve the replication log on an internal gRPC server. This change allows users to view the replication log without having to access the underlying infrastructure. The previous version of Dozer stored the replication log in a file. This meant that users had to access the file to view the replication log. This PR changes how app and api servers communicate. The "internal server" that's used for app api communication was rewritten. Now the internal server has two sets of APIs:
    • describe_build: returns code that's being used by this app server
    • describe_storage and get_log: returns data that are being generated by this app server.
  • Use a different slot name every time when Dozer connects to Postgres (#1742): A bug was fixed in this edit where Dozer would always use the same slot name when connecting to PostgreSQL. This could cause problems if Dozer was running multiple instances on the same server. The bug was caused by a typo in the code. The typo was fixed and Dozer now uses a different slot name for each instance. For a more in-depth understanding, we recommend exploring the documents.
  • Synchronize schema change (#1744): A bug was fixed in this edit where Dozer was not synchronizing schema changes between the source and target databases. This could cause problems if the schema of the source database was changed and Dozer was not aware of the change. The bug was caused by a missing call to the sync_schema() function. The call was added and Dozer now synchronizes schema changes between the source and target databases. For more comprehensive information, consult the detailed documentation.
  • Don't serialize the app name if it is missing (#1746): This edit fixed a bug where Dozer would serialize an empty string as the app name if the app name was not specified in the configuration file. This could cause problems if the app name was not specified and Dozer was trying to log the app name. The bug was caused by a typo in the code. The typo was fixed and Dozer now does not serialize an empty string as the app name if the app name is not specified in the configuration file. Here is a sample configuration of " Don't serialize the app name if it is missing":
#[derive(Serialize, Deserialize, PartialEq, Eq, Clone, prost::Message)]
/// The configuration for the app
pubstructConfig {
#[serde(skip\_serializing\_if = "String::is\_empty")]
#[prost(string, tag = "2")]
/// name of the app
pub app\_name: String,
  • Pin toolchain version to RUSTC_VERSION: 1.70.0 (#1748): This edit pinned the toolchain version to RUSTC_VERSION: 1.70.0 instead of stable. This change makes it easier to reproduce builds and ensures that Dozer is always built with the same version of the Rust toolchain. The line for aarch64 release build rustup target was added: aarch64-unknown-linux-gnu Our goal is to empower users to efficiently interact with our system, boosting productivity and satisfaction.
  • Added toolchain toml to pin version (#1749): This chore added a toolchain.toml file to the project. This file specifies the version of the Rust toolchaVersion 0.1.28 has been successfully prepared for release. This milestone represents significant progress in our product's development, including important updates and enhancements. Our team worked diligently to finalize the codebase, conduct thorough testing, and ensure the release's stability and reliability. Along with bug fixes, we incorporated new features and functionalities based on user feedback and market demands. With the preparation of v0.1.28 complete, our focus is on delivering an improved user experience and valuable enhancements.in that should be used to build Dozer. The addition of the toolchain.toml file makes it easier to pin the toolchain version and ensures that Dozer is always built with the same version of the Rust toolchain. Here is a sample configuration of the Added toolchain toml to pin version:
[toolchain]
channel = "1.70.0"
components = ["clippy", "rustfmt"]
  • MIN_VALUE() implementation + unit tests (#1740): This featadded the MIN_VALUE() function. This function returns the minimum value of two expressions. The MIN_VALUE() function is a valuable addition to Dozer. It makes it easier for users to calculate the minimum value of two expressions. For a more detailed understanding, we recommend exploring our extensive documents.
  • Log reader was duplicating records that were persisted (#1750): A bug was fixed in this edit where the log reader was duplicating records that were persisted. This bug could cause problems if the log reader was used to read a large number of records. The bug was caused by a typo in the code. The typo was fixed and the log reader no longer duplicates records that are persisted. Please click here to see the edits made in the code.
  • Prepared v0.1.28 (#1751): Version 0.1.29 is ready for release, representing significant progress in Dozer's development. Our team worked diligently to finalize the codebase, conduct thorough testing, and ensure stability. Along with bug fixes, we incorporated new features and functionalities based on user feedback and market demands. Our focus now shifts to delivering an improved user experience and valuable enhancements.

We are extremely proud of the progress made in our latest release, v0.1.29. Our team is fully committed to improving Dozer, and we sincerely appreciate the valuable feedback and contributions from our incredible community. Let's continue our collaborative journey of innovation together! 🚀

Dozer v0.1.29 is now available. For more detailed information, please refer to the release notes.

Looking Forward 🌈

As we continue to improve Dozer, our commitment lies in expanding its features and capabilities. Your feedback plays a vital role in our development process, and we encourage you to share your thoughts and ideas. You can open an issue for feature requests or can start a discussion thread in our GitHub Q&A category for feedback. Together, we have the power to elevate Dozer to its maximum potential and take it to new heights.

Full Changelog:

Contact us 📬