Broadcasting, Measurement, Metrics and Snapshot

Optimize checkpointing in your Amazon Managed Service for Apache Flink applications with buffer debloating and unaligned checkpoints – Part 2

AWS Big Data

SEPTEMBER 14, 2023

We’ve already discussed how checkpoints, when triggered by the job manager, signal all source operators to snapshot their state, which is then broadcasted as a special record called a checkpoint barrier. When barriers from all upstream partitions have arrived, the sub-task takes a snapshot of its state.

Snapshot

Snapshot Broadcasting Optimization Management

Optimize checkpointing in your Amazon Managed Service for Apache Flink applications with buffer debloating and unaligned checkpoints – Part 1

AWS Big Data

SEPTEMBER 14, 2023

Each of the distributed components of an application asynchronously snapshots its state to an external persistent datastore. The challenge is taking snapshots guaranteeing exactly-once consistency. When a downstream operator’s sub-task receives all checkpoint barriers from all input channels, it starts snapshotting its state.

Optimization

Optimization Snapshot Management Broadcasting

Data Leaders Brief

Optimize checkpointing in your Amazon Managed Service for Apache Flink applications with buffer debloating and unaligned checkpoints – Part 2

Optimize checkpointing in your Amazon Managed Service for Apache Flink applications with buffer debloating and unaligned checkpoints – Part 1

Webinars

Stay Connected