AWS Big Data

MARCH 2, 2023

Iceberg takes advantage of the rich metadata it captures at write time and facilitates techniques such as scan planning, partitioning, pruning, and column-level stats such as min/max values to skip data files that don’t have match records. It first uses the manifest list, which acts as an index of the manifest files.

Data Lake 116

Data Lake Data Processing Metadata Snapshot 116

AWS Big Data

MARCH 27, 2023

Register the schema To connect Athena to Confluent, the connector needs the schema of the topic to be registered in the AWS Glue Schema Registry , which Athena uses for query planning. transactions" WHERE product_category='Kids' In the Connection details section, choose Create Lambda function. transactions_db"."transactions"

Visualization 74

Visualization Data Lake Interactive Data-driven 74

AWS Big Data

MAY 24, 2023

During a planned or unplanned regional traffic disruption, failover controls let you control failover between buckets in different Regions and accounts within minutes. Run the following Spark commands in your PySpark notebook: df = spark.read.parquet("s3://amazon-reviews-pds/parquet/product_category=Electronics/*.parquet")

Data Lake 83

Data Lake Snapshot Metadata Optimization 83