AWS Big Data

JULY 3, 2023

With scalable metadata indexing, Apache Iceberg is able to deliver performant queries to a variety of engines such as Spark and Athena by reducing planning time. To avoid look-ahead bias in backtesting, it’s essential to create snapshots of the data at different points in time. Tag this data to preserve a snapshot of it.

Snapshot 84

Snapshot Data Lake Testing Strategy 84

AWS Big Data

JANUARY 24, 2023

The utility for cloning and experimentation is available in the open-sourced GitHub repository. This solution only replicates metadata in the Data Catalog, not the actual underlying data. Lake Formation permissions In Lake Formation, there are two types of permissions: metadata access and data access.

Data Architecture 67

Data Architecture Metadata Data Lake Snapshot 67

AWS Big Data

MAY 24, 2023

The following examples are also available in the sample notebook in the aws-samples GitHub repo for quick experimentation. In that case, we have to query the table with the snapshot-id corresponding to the deleted row. We expire the old snapshots from the table and keep only the last two.

Data Lake 85

Data Lake Snapshot Metadata Optimization 85

AWS Big Data

JULY 20, 2023

Additionally, partition evolution enables experimentation with various partitioning strategies to optimize cost and performance without requiring a rewrite of the table’s data every time. Metadata tables offer insights into the physical data storage layout of the tables and offer the convenience of querying them with Athena version 3.

Data Lake 88

Data Lake Analytics Snapshot Optimization 88