AWS Big Data

SEPTEMBER 14, 2023

We’ve already discussed how checkpoints, when triggered by the job manager, signal all source operators to snapshot their state, which is then broadcasted as a special record called a checkpoint barrier. When barriers from all upstream partitions have arrived, the sub-task takes a snapshot of its state.

Snapshot 91

Snapshot Broadcasting Optimization Management 91

AWS Big Data

MARCH 18, 2024

The DataStream API now supports features like side outputs and broadcast state, and gaps on windowing API have been closed. where the operator state couldn’t be properly restored when snapshot compression is enabled. Also, we recommend testing the updated application before proceeding with the update.

Management 103

Management Snapshot Broadcasting Optimization 103

AWS Big Data

SEPTEMBER 14, 2023

Each of the distributed components of an application asynchronously snapshots its state to an external persistent datastore. The challenge is taking snapshots guaranteeing exactly-once consistency. When a downstream operator’s sub-task receives all checkpoint barriers from all input channels, it starts snapshotting its state.

Optimization 83

Optimization Snapshot Management Broadcasting 83