AWS Big Data

OCTOBER 3, 2023

An in-place migration can be performed in either of two ways: Using add_files : This procedure adds existing data files to an existing Iceberg table with a new snapshot that includes the files. Unlike migrate or snapshot, add_files can import files from a specific partition or partitions and doesn’t create a new Iceberg table.

Data Lake 102

Data Lake Metadata Snapshot Recreation/Entertainment 102

AWS Big Data

OCTOBER 3, 2023

For our testing, we generated about 58,176 small objects with total size of 2 GB. For running the Amazon EMR tests, we used Amazon EMR version emr-6.11.0 Check the snapshot table to see that a new snapshot is created for the table with the operation replace. with Spark 3.3.2, and JupyterEnterpriseGateway 2.6.0.

Optimization 109

Optimization Snapshot Data Lake Metadata 109

AWS Big Data

MARCH 2, 2023

Whenever there is an update to the Iceberg table, a new snapshot of the table is created, and the metadata pointer points to the current table metadata file. At the top of the hierarchy is the metadata file, which stores information about the table’s schema, partition information, and snapshots.

Data Lake 116

Data Lake Data Processing Metadata Snapshot 116

Cloudera

FEBRUARY 8, 2024

Overview This blog post describes support for materialized views for the Iceberg table format. Create Iceberg materialized view For the examples in this blog, we will use three tables from the TPC-DS dataset as our base tables: store_sales, customer and date_dim. Both full and incremental rebuild of the materialized view are supported.

Snapshot 88

Snapshot Metadata Cost-Benefit Data Warehouse 88

AWS Big Data

MAY 24, 2023

Update your-iceberg-storage-blog in the following configuration with the bucket that you created to test this example. S3FileIO", "spark.sql.catalog.dev.warehouse":"s3://<your-iceberg-storage-blog>/iceberg/", "spark.sql.catalog.dev.s3.write.tags.write-tag-name":"created",

Data Lake 81

Data Lake Snapshot Metadata Optimization 81

AWS Big Data

MARCH 28, 2023

Test SCD Type 2 implementation With the infrastructure in place, you’re ready to test out the overall solution design and query historical records from the employee dataset. This post is designed to be implemented for a real customer use case, where you get full snapshot data on a daily basis.

Data Lake 70

Data Lake Testing Snapshot Sales 70

Laminar Security

NOVEMBER 9, 2023

Malicious actors came out swinging at the start of 2023, and they aren’t slowing down any time soon. An industry-accepted framework can serve as a litmus test to ensure that your chosen platform covers the most critical facets of data security and keeps bad actors at bay. Facilitating proactive recovery testing.

Strategy 88

Strategy Risk Testing Recreation/Entertainment 88