Skip to main content

Simplifying Real-Time Ethereum Dashboards with Dozer

· 11 min read
Aarya

Welcome to an exciting journey into the world of real-time Ethereum data integration with Dozer. In this comprehensive tutorial, we will guide you through the seamless configuration of Dozer, enabling you to ingest real-time Ethereum data, merge it with off-chain information, and effortlessly access it through queryable APIs. Our focus will be on the ever-popular USDT Token, as we monitor the Ethereum Chain for USDT Transfer Events.

From the initial environment setup to querying data using Dozer and culminating in the creation of a stunning real-time Ethereum dashboard showcasing valuable analytics, this guide covers it all. Dozer's power for data transformation and accessibility is within your grasp, and we're here to make the journey smooth and exciting.

Join us as we unlock the potential of Dozer, turning Ethereum data into a valuable resource for your real-time needs.

What is Dozer?

Dozer stands as a robust open-source data API backend, streamlining the intricate task of ingesting real-time data from diverse sources and delivering it effortlessly through REST and gRPC endpoints. Its versatility is evident in its support for a vast array of data sources, ranging from databases to object storage services and beyond. Furthermore, Dozer simplifies the development process with its built-in Dozer Live playground, accessible through the CLI. This feature serves as a valuable tool, aiding developers in seamless debugging and application development.You can learn more about Dozer from its official documentation and its GitHub repository.

Why to use Dozer?

Dozer is a powerful tool for building data products the easy way. It makes it easy to ingest data from various sources and expose it as queryable APIs through a single configuration file. With a powerful streaming SQL engine and caching fully built in Rust, Dozer empowers developers to build low-latency APIs with complex transformations.

In this particular case Dozer will use its Ethereum connector to continuously monitor and fetch data from the USDT Transfer Events. Dozer also provides the ability to merge off-chain data with on-chain data, which can be used to identify buyers and sellers in real-time transactions and flag any suspicious Ethereum addresses involved in these exchanges, additionally historical data stored in any database can be merged with the real-time data using UNION transformations. With the OnEvent endpoint, changes in the data sources are immediately reflected on this endpoint. Using Dozer, analytics can be performed with dashboards that update not in days or hours, but in seconds.

Prerequisites

Before we begin, make sure you have the following:

  • An Ethereum Web Sockets URL. Get one from Infura.
  • Dozer installed. For Ethereum, you need to build and run Dozer from source. For more installation instructions, visit the Dozer documentation.
  • Basic knowledge of SQL for writing data transformation queries.

Understanding The Data

In this tutorial, we will utilise the WebSockets URL provided by Infura to monitor the Tether(USDT) smart contract on Ethereum. Ethereum WebSockets is a protocol that allows for real-time, bidirectional communication between a client and a server on the Ethereum blockchain. It can be used to stream updates about the state of the blockchain and the transactions of NFTs or other ERC20 Tokens.

Tether (USDT) is a stablecoin in the cryptocurrency world, designed to maintain a value pegged to traditional fiat currencies like the US Dollar. It provides stability in a volatile market, making it a popular choice for traders and investors looking to safeguard their assets while navigating the cryptocurrency space. There are a variety of potential applications for using WebSockets to query data related to ERC-20 tokens. For example, you could use it to monitor the price of a token, track the number of transactions, or even create a real-time Ethereum dashboard to display the data.

Here is a sample connection block that we will use in this tutorial, to specify the connection to Ethereum:

- config: !Ethereum
provider: !Log
wss_url: {{ETH_WSS_URL}}
contracts:
- name: usdt
address: 0xdAC17F958D2ee523a2206206994597C13D831ec7
abi: >
[{"constant":true,"inputs":[],"name":"name","outputs":[{"name":"","type":"string"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_upgradedAddress","type":"address"}],"name":"deprecate","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":false,"inputs":[{"name":"_spender","type":"address"},{"name":"_value","type":"uint256"}],"name":"approve","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"deprecated","outputs":[{"name":"","type":"bool"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_evilUser","type":"address"}],"name":"addBlackList","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"totalSupply","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_from","type":"address"},{"name":"_to","type":"address"},{"name":"_value","type":"uint256"}],"name":"transferFrom","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"upgradedAddress","outputs":[{"name":"","type":"address"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"","type":"address"}],"name":"balances","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"decimals","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"maximumFee","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"_totalSupply","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[],"name":"unpause","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[{"name":"_maker","type":"address"}],"name":"getBlackListStatus","outputs":[{"name":"","type":"bool"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"","type":"address"},{"name":"","type":"address"}],"name":"allowed","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"paused","outputs":[{"name":"","type":"bool"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"who","type":"address"}],"name":"balanceOf","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[],"name":"pause","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"getOwner","outputs":[{"name":"","type":"address"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"owner","outputs":[{"name":"","type":"address"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"symbol","outputs":[{"name":"","type":"string"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_to","type":"address"},{"name":"_value","type":"uint256"}],"name":"transfer","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":false,"inputs":[{"name":"newBasisPoints","type":"uint256"},{"name":"newMaxFee","type":"uint256"}],"name":"setParams","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":false,"inputs":[{"name":"amount","type":"uint256"}],"name":"issue","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":false,"inputs":[{"name":"amount","type":"uint256"}],"name":"redeem","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[{"name":"_owner","type":"address"},{"name":"_spender","type":"address"}],"name":"allowance","outputs":[{"name":"remaining","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"basisPointsRate","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"","type":"address"}],"name":"isBlackListed","outputs":[{"name":"","type":"bool"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_clearedUser","type":"address"}],"name":"removeBlackList","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"MAX_UINT","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"newOwner","type":"address"}],"name":"transferOwnership","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":false,"inputs":[{"name":"_blackListedUser","type":"address"}],"name":"destroyBlackFunds","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"inputs":[{"name":"_initialSupply","type":"uint256"},{"name":"_name","type":"string"},{"name":"_symbol","type":"string"},{"name":"_decimals","type":"uint256"}],"payable":false,"stateMutability":"nonpayable","type":"constructor"},{"anonymous":false,"inputs":[{"indexed":false,"name":"amount","type":"uint256"}],"name":"Issue","type":"event"},{"anonymous":false,"inputs":[{"indexed":false,"name":"amount","type":"uint256"}],"name":"Redeem","type":"event"},{"anonymous":false,"inputs":[{"indexed":false,"name":"newAddress","type":"address"}],"name":"Deprecate","type":"event"},{"anonymous":false,"inputs":[{"indexed":false,"name":"feeBasisPoints","type":"uint256"},{"indexed":false,"name":"maxFee","type":"uint256"}],"name":"Params","type":"event"},{"anonymous":false,"inputs":[{"indexed":false,"name":"_blackListedUser","type":"address"},{"indexed":false,"name":"_balance","type":"uint256"}],"name":"DestroyedBlackFunds","type":"event"},{"anonymous":false,"inputs":[{"indexed":false,"name":"_user","type":"address"}],"name":"AddedBlackList","type":"event"},{"anonymous":false,"inputs":[{"indexed":false,"name":"_user","type":"address"}],"name":"RemovedBlackList","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"owner","type":"address"},{"indexed":true,"name":"spender","type":"address"},{"indexed":false,"name":"value","type":"uint256"}],"name":"Approval","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"from","type":"address"},{"indexed":true,"name":"to","type":"address"},{"indexed":false,"name":"value","type":"uint256"}],"name":"Transfer","type":"event"},{"anonymous":false,"inputs":[],"name":"Pause","type":"event"},{"anonymous":false,"inputs":[],"name":"Unpause","type":"event"}]

name: eth_conn

To monitor smart contracts on Ethereum, all we need to do is provide the smart contract address ,and its ABI (Application Binary Interface). The ABI is a JSON file that describes the functions and events of the smart contract. It is used to decode the data emitted by the smart contract and to call its functions. The ABI is available on the Etherscan website for most smart contracts.

In this case:

We will be using the Transfer Event emitted by USDT (or any ERC-20 token for that matter), whenever a transaction occurs. The Transfer Event contains the following fields:

  • from : The address of the sender.
  • to : The address of the receiver.
  • value : The amount of tokens transferred. In the case of USDT, this is the amount of USDT transferred (utilising 6 decimal places).

Adding Transformations

Now, that the connection has been established, we can look to add transformations to generate useful insights.

For that, first we need to specify the sources that we are listening from.

sources:
- name: usdt_Transfer
table_name: usdt_Transfer
connection: eth_conn

NOTE: The table name can be generated as: {contractname}{event_name}


sql: |
WITH data AS (
SELECT value, NOW() AS currtime
FROM usdt_Transfer
)
SELECT
t.window_start AS start, t.window_end AS end, (SUM(t.value))/1e6 AS traded_volume, AVG(t.value)/1e6 AS averg_transaction, MAX(t.value)/1e6 AS max_transaction, MIN(t.value)/1e6 AS min_transaction
INTO sum
FROM TUMBLE(data, currtime,'24 HOURS') t
GROUP BY t.window_start, t.window_end;

Next, we need to specify the transformations that we want to run on the data. In this case, we are using a TUMBLE function to aggregate the data into 24 hour windows.

In Dozer, the TUMBLE() function serves to create distinct and non-overlapping time windows for streaming data. Each window has a defined duration, and data falling within each window is aggregated. These time windows prove valuable when conducting recurrent aggregations or computations on streaming data.

As the on-chain data does not contain a timestamp, we use the NOW() function to generate a timestamp for each event. This timestamp is then used to create the time windows.

We are also converting the value USDT by dividing it by 1e6 (as USDT uses 6 decimal places to specify its amount). We perform 4 aggregations on the value field: SUM, AVG, MAX, and MIN . All the aggregations happen in real time, and the results are stored in a table called sum.

This helps us to get a better understanding of the volume of USDT being transferred on the Ethereum blockchain. We can also use this data to create a real-time dashboard to display the results.

Merging Off Chain Data

A standout feature of Dozer is its ability to seamlessly merge data from diverse sources and present it as real-time queryable APIs. The system ensures data freshness by constantly updating the cache with the latest information from these sources, guaranteeing that users always access the most current data through the APIs.

Now, let's dive into a practical use case. We aim to identify buyers and sellers engaged in USDT transactions. Utilizing a prebuilt CSV dataset, compiled from web scraping efforts, we've associated Ethereum Addresses with user names and assessed their legitimacy. This dataset will enable us to dynamically label buyers and sellers in real-time transactions and flag any suspicious Ethereum addresses involved in these exchanges.

To merge this off-chain data with the on-chain data, we need to add the sources and transformations to our configuration.


#Connections
- config : !LocalStorage
details:
path: .
tables:
- !Table
name: eth_addresses
config: !CSV
path: data
extension: .csv
name: local

#Sources
sources:
- name: off_chain
table_name: eth_addresses
connection: local

#Transformations
sql: |
SELECT e.Address AS seller_address,e.Name as Name, e.Label AS Authenticity
INTO identified_sellers
FROM
usdt_Transfer u
JOIN off_chain e
ON u.from = e.Address;

SELECT e.Address AS buyer_address,e.Name as Name, e.Label AS Authenticity
INTO identified_buyers
FROM
usdt_Transfer u
JOIN off_chain e
ON u.to = e.Address;

We are using a JOIN to merge the on-chain data with the off-chain data. The JOIN is performed on the from and to fields of the on-chain data and the Address field of the off-chain data. The JOIN is performed twice, once for the from field and once for the to field. This allows us to identify both the buyer and the seller in each transaction.

The results of the JOIN are stored in two tables: identified_sellers and identified_buyers . These tables contain the Ethereum addresses of the buyers and sellers, along with their names and authenticity labels.

Endpoints and APIs

Lastly, we will specify the endpoints and APIs that we want to expose to the user.


#Endpoints
endpoints:

# USDT Transfers
- name: transfers
table_name: transfers
path: /transfers

# Aggregated Traded Volume
- name: sum
table_name: sum
path: /sum

# Top Buyers
- name: buyers
table_name: buyers
path: /buyers

# Top Sellers
- name: sellers
table_name: sellers
path: /sellers

# Identified Sellers
- name: identified_sellers
table_name: identified_sellers
path: /identified_sellers

# Identified Buyers
- name: identified_buyers
table_name: identified_buyers
path: /identified_buyers

Dozer offers a powerful capability of seamlessly merging off-chain and on-chain data, resulting in the generation of automatic REST and gRPC APIs based on the endpoint configuration provided in the Dozer configuration.

With these merged data sources, you can harness the full potential of Dozer's capabilities. By configuring the endpoints to align with your specific requirements, you enable the creation of APIs that allow for real-time querying of the combined data. This means you can effortlessly query the integrated data cache using either gRPC or REST, depending on your preference. This flexibility ensures that you can access and retrieve the results of your SQL queries efficiently and effectively, leveraging the richness of both off-chain and on-chain data sources.

Running Dozer

Once the configuration file is set up, we can start Dozer by running the following command in the terminal:

dozer run

This will start Dozer and it will begin ingesting data from the specified Ethereum WebSockets URL and populating the cache. Alternatively, you can also start Dozer Live to see the ingestion progress in real-time.

Run

Querying the Dozer APIs

Dozer generates automatic REST and gRPC APIs based on the endpoint configuration provided. We can now query the Dozer endpoints to get the results of our SQL queries. You can query the cache using gRPC (using typed/common service) or REST.

By default, the gRPC service starts on port 50051 and the REST service starts on port 8080. You can change these ports in the configuration file.

Dozer APIs also support filters for pagination, sorting, and filtering. You can learn more about these filters from the official documentation.

Developing a Real-Time Dashboard

A highly effective application of Dozer is in the creation of real-time dashboards that aggregate data from various sources in real-time. This capability is valuable for analysts and developers as it ensures that they are always working with the most up-to-date data, enhancing their decision-making process.

To build a real-time dashboard with Dozer, we can use the Dozer-React library. This library offers custom hooks that can be integrated into your code to query endpoints. These hooks allow you to send query strings to the endpoints. A query is essentially a JSON object serialized as a string.

Here are some of the key hooks provided by Dozer-React:

  • useDozerEndpointCount: This hook provides the count of records stored in the cache for a specific endpoint.

  • useDozerEndpointQuery: This hook retrieves the list of fields and records for the given endpoint.

  • useDozerEndpointFields: This hook fetches the list of fields available for a particular endpoint.

You can incorporate these hooks into your React code to query endpoints and display the results in your dashboard.

Here are the steps to set up and run the dashboard:

  • Begin by starting the Dozer Live server to commence the ingestion of data from both on-chain and off-chain sources.
dozer live
  • Launch the React Dashboard by running the following command:
npm run dev
  • Access the dashboard at at: http://localhost:5173

Dashboard

You can even combine off-chain data on the dashboard and display it in real time. For example, we can display the buyers and sellers in the dashboard, along with their authenticity labels.

Dashboard

Conclusion

In this tutorial, we've unveiled the incredible potential of implementing visually striking real-time dashboards for the Ethereum blockchain. With Dozer at your disposal, you can effortlessly merge on-chain and off-chain data, bringing a world of valuable insights right to your fingertips. From the inception of real-time data ingestion to the creation of dynamic, eye-catching dashboards, Dozer has proven itself as an invaluable tool for Ethereum developers and analysts. As you embark on your journey with Dozer, consider the vast realm of possibilities that blockchain integration offers. It's a world where you have the tools to explore, innovate, and make informed decisions in real time, making your experience both enriching and exciting. Get ready to explore, innovate, and make informed decisions in real time as you dive into the world of Ethereum with Dozer!