Meta’s Muse Charm might cost $300Amazon Polly launches voice cloning: A unique voice for your brandMicrosoft is killing its great tasks app in favor of its worse tasks appMicrosoft brings AI inference closer to enterprise data with Windows Server 2028Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPodIntroducing the Amazon SageMaker JumpStart Model LeaderboardMeta’s Muse Charm might cost $300Amazon Polly launches voice cloning: A unique voice for your brandMicrosoft is killing its great tasks app in favor of its worse tasks appMicrosoft brings AI inference closer to enterprise data with Windows Server 2028Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPodIntroducing the Amazon SageMaker JumpStart Model Leaderboard
↓ Pull down to refresh stories
Courses Write Login VIVietnamese Store
Verified

How United Airlines streams database changes to Apache Iceberg with Amazon Data Firehose

This post is co-authored with Tunde Fowora, Principal Architect at United Airlines. The part worth reading is how quickly everyday product behavior can shift after an update like this.

This post is co-authored with Tunde Fowora, Principal Architect at United Airlines. United Airlines, headquartered in Chicago, IL, United States, is the world’s largest airline based on revenue passenger miles, according to United Airlines. The useful part sits in the context, the practical impact, and what readers can use to decide the next step.

How United Airlines streams database changes to Apache Iceberg with Amazon Data Firehose

This post is co-authored with Tunde Fowora, Principal Architect at United Airlines. United Airlines, headquartered in Chicago, IL, United States, is the world’s largest airline based on revenue passenger miles, according to United Airlines.

What happened

This post is co-authored with Tunde Fowora, Principal Architect at United Airlines. United Airlines, headquartered in Chicago, IL, United States, is the world’s largest airline based on revenue passenger miles, according to United Airlines.

Where the sources line up

In this post, we explore how United Airlines adopted Apache Iceberg for large-scale data processing while managing near-real-time data synchronization from transactional databases.

Practical impact for readers

United Airlines needed to replicate frequently changing reference data in near-real-time from an operational Amazon Aurora MySQL-Compatible Edition (Aurora MySQL) database to a new cloud-native analytics system built on Apache Iceberg, during a multi-year modernization. Until that migration completed, changes in the source database still had to reach analysts and downstream systems within minutes. The legacy batch process could not keep pace, so teams analyzed market patterns on stale data, which delayed business decisions.

Who should pay attention now

To meet that need, United Airlines built a near-real-time Change Data Capture (CDC) pipeline using Amazon Kinesis Data Streams (Kinesis Data Streams), AWS Lambda (Lambda), Amazon Data Firehose (Firehose), and Apache Iceberg on Amazon Simple Storage Service (Amazon S3) to ingest database change events emitted from a database CDC engine. At United, database CDC is implemented using the AWS Database Migration Service (AWS DMS) CDC engine. Moving to Iceberg removed the scalability limits of the legacy SQL databases. Queries scanning more than 10 million rows no longer hit those bottlenecks, and several hundred business analysts could run complex queries across a broader data domain. In United’s testing, average query latency improved 5x.

What is still unclear

The streaming layer is what keeps this pattern straightforward to operate. Amazon Data Firehose reads the change events from Kinesis Data Streams, writes them to the Iceberg table as Parquet files, applies each insert, update, and delete, and commits a new table snapshot. You don’t build a separate application or manage servers for that delivery step. Kinesis Data Streams can also run in on-demand mode, which scales capacity automatically as change volume grows.

Source notes

Related stories

Apps & SoftwareAmazon Polly launches voice cloning: A unique voice for your brandApps & SoftwareHow to use AI plans without stacking duplicate appsApps & SoftwareMicrosoft is killing its great tasks app in favor of its worse tasks app