AWS Big Data

APRIL 17, 2024

Amazon OpenSearch Service recently introduced the OpenSearch Optimized Instance family (OR1), which delivers up to 30% price-performance improvement over existing memory optimized instances in internal benchmarks, and uses Amazon Simple Storage Service (Amazon S3) to provide 11 9s of durability.

Optimization 116

Optimization Snapshot Metadata Cost-Benefit 116

AWS Big Data

FEBRUARY 1, 2023

This introduces the need for both polling and pushing the data to access and analyze in near-real time. From an operational standpoint, we designed a new shared responsibility model for data ingestion using AWS Glue instead of internal services (REST APIs) designed on Amazon EC2 to extract the data.

Optimization 52

Optimization Forecasting Data Lake Metadata 52

AWS Big Data

NOVEMBER 22, 2023

Hudi provides tables , transactions , efficient upserts and deletes , advanced indexes , streaming ingestion services , data clustering and compaction optimizations, and concurrency control , all while keeping your data in open source file formats. This post demonstrates how this new capability to crawl Hudi tables works.

Data Lake 95

Data Lake Snapshot Metadata Optimization 95

Cloudera

MARCH 23, 2022

Figure 1: Apache Iceberg fits the next generation data architecture by abstracting storage layer from analytics layer while introducing net new capabilities like time-travel and partition evolution. #1: Apache Iceberg enables seamless integration between different streaming and processing engines while maintaining data integrity between them.

Metadata 90

Metadata Data Architecture Machine Learning Cost-Benefit 90

AWS Big Data

JULY 20, 2023

Determining optimal table partitioning Determining optimal partitioning for each table is very important in order to optimize query performance and minimize the impact on teams querying the tables when partitioning changes. The following diagram illustrates the solution architecture. Orca addressed this in several ways.

Data Lake 88

Data Lake Analytics Snapshot Optimization 88

AWS Big Data

JUNE 12, 2023

Acting as a bridge between producer and consumer apps, it enforces the schema, reduces the data footprint in transit, and safeguards against malformed data. AWS Glue is an ideal solution for running stream consumer applications, discovering, extracting, transforming, loading, and integrating data from multiple sources.

Management 80

Management Metadata Testing Internet of Things 80

AWS Big Data

NOVEMBER 29, 2023

The dbt-glue adapter democratized access for dbt users to data lakes, and enabled many users to effortlessly run their transformation workloads on the cloud with the serverless data integration capability of AWS Glue. The team uses dbt-glue to build a transformed gold model optimized for business intelligence (BI).

Data Lake 106

Data Lake Management Metrics Data Warehouse 106