Skip to main content

This week at Dozer #5

ยท 3 min read

๐ŸŒŸ Welcome to this week's update on dozer! Dozer is excited to announce the release of version v0.1.17, which brings a host of new features and improvements. Our team has been working tirelessly to enhance the overall functionality, and we can't wait for you to experience all the new features. Here are the updates for this week.

Key enhancement of v0.1.17 ๐Ÿš€:โ€‹

Switched to Apache License #1404 ๐Ÿ“œ:โ€‹

In this release, Dozer has switched to the Apache License (The 2.0 version), making is 100% Open Source. This change provides a more permissive and flexible licensing option, encouraging collaboration and adoption by a wider range of developers and organizations.

Asyncify Connectors #1409 โšก:โ€‹

In this release, connectors have been refactored to work asynchronously, leading to performance improvements and smoother execution.

SQL Logic Test Framework #1326 ๐Ÿงช:โ€‹

A new SQL Logic Test Framework has been introduced, allowing for more comprehensive testing of Dozer's SQL capabilities.

APIs for Monitoring and Status Updates #1329 ๐Ÿ“ˆ:โ€‹

We've added APIs for monitoring and status updates, enabling better insight into Dozer's performance and health.

Graceful API Thread Shutdown #1416 ๐Ÿ› ๏ธ:โ€‹

The API thread now shuts down gracefully, preventing potential issues with abrupt termination.

Dozer Lambda Functions: A Glimpse into the Future ๐Ÿš€๐Ÿ”ฎ #1424, #1426 ๐Ÿ๐ŸŒ:โ€‹

We're excited to share that we've released JavaScript and Python bindings for reading Dozer logs. These bindings will be part of our Dozer Lambda runtime, allowing you to create Lambda functions in JS and Python that react to Dozer events. Just imagine triggering a Twilio notification based on a SQL statement (data change condition)! This feature is currently experimental, and we'll publish a complete article with use cases and samples soon.

state diagram of dozer lambda runtime on data change condition

Note: DCC stands for Data Change Condition

Here's an example for JavaScript:

const dozer = require('..');

async function main() {
runtime = dozer.Runtime();
reader = await runtime.create_reader('../.dozer/pipeline', 'trips');

for (let i = 0; i < 10; ++i) {
let op = await reader.next_op();



And for Python:

import dozer_log

async def main():
reader = await'../.dozer/pipeline', 'trips')
for _ in range(10):
data = await reader.next_op()

if __name__ == '__main__':
import asyncio

In these examples, the console.log() and print() statements can be replaced with user-defined code.

Other Improvements & Fixes ๐Ÿ”งโ€‹

  • Fixed e2e tests writing to directories that are not removed in #1407
  • Fixed type validation in #1405
  • Replaced std::thread::sleep with tokio::time::sleep in #1413
  • Consolidated to a single Tokio runtime for everything in #1417
  • Used the correct runtime for internal pipeline thread in #1418
  • Refactored as dozer-log crate in #1420
  • Used proper error message for PostgreSQL replication slot creation in #1422
  • Stored schema of sinks separately in #1423
  • Added deb generation to release process in #1432
  • Removed unnecessary workspace.exclude in Cargo.toml in #1430
  • Bumped h2 from 0.3.16 to 0.3.17 in #1427

Dozer v0.1.17 is available now. Check out the release notes here.

Looking Forward ๐ŸŒˆ

As we continue to make strides in improving Dozer, we remain committed to enhancing its features and functionality. Your feedback is invaluable to us, so please don't hesitate to reach out with any suggestions or ideas. Together, we can make Dozer the best it can be!

Full Changelog:โ€‹

Contact us ๐Ÿ“ฌโ€‹