AWS Big Data

OCTOBER 3, 2023

Iceberg tables store metadata in manifest files. As the number of data files increase, the amount of metadata stored in these manifest files also increases, leading to longer query planning time. The query runtime also increases because it’s proportional to the number of data or metadata file read operations.

Optimization 112

Optimization Snapshot Data Lake Metadata 112

AWS Big Data

JUNE 5, 2023

If you also needed to preserve the history of DAG runs, you had to take a backup of your metadata database and then restore that backup on the newly created environment. Amazon MWAA manages the entire upgrade process, from provisioning new Apache Airflow versions to upgrading the metadata database.

Snapshot 79

Snapshot Metadata Testing Data-driven 79

AWS Big Data

JANUARY 8, 2024

In the subsequent post in our series, we will explore the architectural patterns in building streaming pipelines for real-time BI dashboards, contact center agent, ledger data, personalized real-time recommendation, log analytics, IoT data, Change Data Capture, and real-time marketing data.

Analytics 115

Analytics IoT Data-driven Snapshot 115

Cloudera

NOVEMBER 13, 2020

Operational, Cybersecurity, and IoT reporting where the current point in time state of an individual or single device needs to be analyzed. . Metadata Caching. The new Catalog design means that Impala coordinators will only load the metadata that they need instead of a full snapshot of all the tables. More on this below.

Optimization 143

Optimization Metadata Statistics Cost-Benefit 143

AWS Big Data

JUNE 12, 2023

This data can come from a diverse range of sources, including Internet of Things (IoT) devices, user applications, and logging and telemetry information from applications, to name a few. After the processed data is stored in Amazon S3, we create an AWS Glue crawler to create a Data Catalog table that acts as a metadata layer for the data.

Management 82

Management Metadata Testing Internet of Things 82