AWS Big Data

OCTOBER 3, 2023

This means the data files in the data lake aren’t modified during the migration and all Apache Iceberg metadata files (manifests, manifest files, and table metadata files) are generated outside the purview of the data. In this method, the metadata are recreated in an isolated environment and colocated with the existing data files.

Data Lake 100

Data Lake Metadata Snapshot Recreation/Entertainment 100

AWS Big Data

MARCH 2, 2023

Apache Iceberg is an open table format for very large analytic datasets, which captures metadata information on the state of datasets as they evolve and change over time. Apache Iceberg addresses customer needs by capturing rich metadata information about the dataset at the time the individual data files are created.

Data Lake 116

Data Lake Data Processing Metadata Snapshot 116

Cloudera

FEBRUARY 14, 2024

Problem with too many snapshots Everytime a write operation occurs on an Iceberg table, a new snapshot is created. A bloated metadata.json file could increase both read/write times because a large metadata file needs to be read/written every time. You could also change the isolation level to snapshot isolation.

Strategy 56

Strategy Optimization Snapshot Metadata 56

Cloudera

JULY 14, 2023

They also provide a “ snapshot” procedure that creates an Iceberg table with a different name with the same underlying data. You could first create a snapshot table, run sanity checks on the snapshot table, and ensure that everything is in order. Hive creates Iceberg’s metadata files for the same exact table.

Snapshot 69

Snapshot Metadata Data Warehouse Testing 69

Cloudera

FEBRUARY 8, 2024

The snapshotId of the source tables involved in the materialized view are also maintained in the metadata. Subsequently, these snapshot IDs are used to determine the delta changes that should be applied to the materialized view rows. Furthermore, it is partitioned on the d_year column.

Snapshot 87

Snapshot Metadata Cost-Benefit Data Warehouse 87

AWS Big Data

SEPTEMBER 13, 2023

The File Manager Lambda function consumes those messages, parses the metadata, and inserts the metadata to the DynamoDB table odpf_file_tracker. Current snapshot – This table in the data lake stores latest versioned records (upserts) with the ability to use Hudi time travel for historical updates.

Data Lake 89

Data Lake Data Processing Metadata Snapshot 89

Cloudera

JULY 13, 2023

A range of Iceberg table analysis such as listing table’s data file, selecting table snapshot, partition filtering, and predicate filtering can be delegated through Iceberg Java API instead, obviating the need for each query engine to implement it themself. The data files and metadata files in Iceberg format are immutable.

Metadata 69

Metadata Snapshot Data Warehouse Statistics 69