Broadcasting, Measurement, Optimization and Snapshot

Optimize checkpointing in your Amazon Managed Service for Apache Flink applications with buffer debloating and unaligned checkpoints – Part 2

AWS Big Data

SEPTEMBER 14, 2023

We’ve already discussed how checkpoints, when triggered by the job manager, signal all source operators to snapshot their state, which is then broadcasted as a special record called a checkpoint barrier. When barriers from all upstream partitions have arrived, the sub-task takes a snapshot of its state.

Snapshot

Snapshot Broadcasting Optimization Management

Optimize checkpointing in your Amazon Managed Service for Apache Flink applications with buffer debloating and unaligned checkpoints – Part 1

AWS Big Data

SEPTEMBER 14, 2023

Internally, Apache Flink uses clever mechanisms to maintain exactly-once state consistency, while also optimizing for throughput and reduced latency. Each of the distributed components of an application asynchronously snapshots its state to an external persistent datastore. The default behavior works well for most use cases.

Optimization

Optimization Snapshot Management Broadcasting

Data Leaders Brief

Optimize checkpointing in your Amazon Managed Service for Apache Flink applications with buffer debloating and unaligned checkpoints – Part 2

Optimize checkpointing in your Amazon Managed Service for Apache Flink applications with buffer debloating and unaligned checkpoints – Part 1

Webinars

Stay Connected