AWS Big Data

APRIL 3, 2024

Apache Iceberg offers integrations with popular data processing frameworks such as Apache Spark, Apache Flink, Apache Hive, Presto, and more. AWS provides integrations for various AWS services with Iceberg tables as well, including AWS Glue Data Catalog for tracking table metadata.

Data Lake 91

Data Lake Snapshot Metadata Data Architecture 91

AWS Big Data

OCTOBER 3, 2023

With changing use cases, customers are looking for ways to not only move new or incremental data to data lakes as transactions, but also to convert existing data based on Apache Parquet to a transactional format. In this method, the metadata are recreated in an isolated environment and colocated with the existing data files.

Data Lake 98

Data Lake Metadata Snapshot Recreation/Entertainment 98

IBM Big Data Hub

JULY 6, 2022

With traditional approaches , data issues are reported by data users as they try to access and use the data and may take weeks to fix, if they’re found at all. starts at the data source, collecting data pipeline metadata across key solutions in the modern data stack like Airflow, dbt, Databricks and many more.

Metadata 57

Metadata Data Quality Snapshot Cost-Benefit 57

AWS Big Data

APRIL 17, 2024

In this post, we discuss how the reimagined data flow works with OR1 instances and how it can provide high indexing throughput and durability using a new physical replication protocol. We also dive deep into some of the challenges we solved to maintain correctness and data integrity.

Optimization 110

Optimization Snapshot Metadata Cost-Benefit 110

AWS Big Data

NOVEMBER 22, 2023

Apache Hudi is an open table format that brings database and data warehouse capabilities to data lakes. Apache Hudi helps data engineers manage complex challenges, such as managing continuously evolving datasets with transactions while maintaining query performance.

Data Lake 88

Data Lake Snapshot Metadata Optimization 88

AWS Big Data

FEBRUARY 1, 2023

We also used AWS Lambda for data processing. To further optimize and improve the developer velocity for our data consumers, we added Amazon DynamoDB as a metadata store for different data sources landing in the data lake. Clients access this data store with an API’s.

Optimization 52

Optimization Forecasting Data Lake Metadata 52

Cloudera

MARCH 23, 2022

Figure 1: Apache Iceberg fits the next generation data architecture by abstracting storage layer from analytics layer while introducing net new capabilities like time-travel and partition evolution. #1: Apache Iceberg enables seamless integration between different streaming and processing engines while maintaining data integrity between them.

Metadata 93

Metadata Data Architecture Machine Learning Cost-Benefit 93