By WalkingTree March 16, 2021
If you are familiar with databases and, data warehouses, then you must have come across the term ETL (Extract, Transform, and Load). It’s a process wherein data is gathered from databases and loaded into another data warehouse such as Amazon Redshift.
Data teams are constantly in sync with different teams for collecting data, which can be hectic sometimes. This task can be replaced with IDT (Intentional Data Transfer). The need for ETL arises only when teams build their database without keeping downstream analytics in mind.
In ITD, you can directly add the logic in the code that first processes the events and emits them in a pub/sub-model. And without any extra effort, the data team can set-up a subscriber process to receive these events. Apart from this, there are other benefits of IDT –
- With IDT, when an event occurs, it will be published with certain fields always present and that is previously agreed upon and documented. Any changes made will not affect the fields in the IDT publisher’s events.
- Events are published immediately as they happen in IDT. Using real-time services like Amazon SNS and Lambda, they can be responded to immediately.
Read on to know more about Intentional Data Transfer.