This post is co-authored with Tunde Fowora, Principal Architect at United Airlines. United Airlines, headquartered in Chicago, IL, United States, is the world’s largest airline based on revenue passenger miles, according to United Airlines.
What happened
This post is co-authored with Tunde Fowora, Principal Architect at United Airlines. United Airlines, headquartered in Chicago, IL, United States, is the world’s largest airline based on revenue passenger miles, according to United Airlines.
Where the sources line up
In this post, we explore how United Airlines adopted Apache Iceberg for large-scale data processing while managing near-real-time data synchronization from transactional databases.
Practical impact for readers
United Airlines needed to replicate frequently changing reference data in near-real-time from an operational Amazon Aurora MySQL-Compatible Edition (Aurora MySQL) database to a new cloud-native analytics system built on Apache Iceberg, during a multi-year modernization. Until that migration completed, changes in the source database still had to reach analysts and downstream systems within minutes. The legacy batch process could not keep pace, so teams analyzed market patterns on stale data, which delayed business decisions.
Who should pay attention now
To meet that need, United Airlines built a near-real-time Change Data Capture (CDC) pipeline using Amazon Kinesis Data Streams (Kinesis Data Streams), AWS Lambda (Lambda), Amazon Data Firehose (Firehose), and Apache Iceberg on Amazon Simple Storage Service (Amazon S3) to ingest database change events emitted from a database CDC engine. At United, database CDC is implemented using the AWS Database Migration Service (AWS DMS) CDC engine. Moving to Iceberg removed the scalability limits of the legacy SQL databases. Queries scanning more than 10 million rows no longer hit those bottlenecks, and several hundred business analysts could run complex queries across a broader data domain. In United’s testing, average query latency improved 5x.
What is still unclear
The streaming layer is what keeps this pattern straightforward to operate. Amazon Data Firehose reads the change events from Kinesis Data Streams, writes them to the Iceberg table as Parquet files, applies each insert, update, and delete, and commits a new table snapshot. You don’t build a separate application or manage servers for that delivery step. Kinesis Data Streams can also run in on-demand mode, which scales capacity automatically as change volume grows.
Latest comments
0No comments yet. You can start the conversation.