AWS Big Data

JULY 26, 2023

Many AWS customers have integrated their data across multiple data sources using AWS Glue , a serverless data integration service, in order to make data-driven business decisions. Are there recommended approaches to provisioning components for data integration?

Data Integration 69

Data Integration Snapshot Testing Visualization 69

AWS Big Data

DECEMBER 13, 2023

Redshift streaming ingestion provides low latency, high-throughput data ingestion, which enables customers to derive insights in seconds instead of minutes. After that, using materialized-view refresh, you can ingest hundreds of megabytes of data per second. This solution uses Amazon Aurora MySQL hosting the example database salesdb.

Data Warehouse 94

Data Warehouse Snapshot Data Processing Management 94

AWS Big Data

OCTOBER 3, 2023

With changing use cases, customers are looking for ways to not only move new or incremental data to data lakes as transactions, but also to convert existing data based on Apache Parquet to a transactional format. In this post, we show you how you can use the Iceberg add_files procedure for an in-place data upgrade.

Data Lake 98

Data Lake Metadata Snapshot Recreation/Entertainment 98

AWS Big Data

APRIL 17, 2024

In this post, we discuss how the reimagined data flow works with OR1 instances and how it can provide high indexing throughput and durability using a new physical replication protocol. We also dive deep into some of the challenges we solved to maintain correctness and data integrity.

Optimization 110

Optimization Snapshot Metadata Cost-Benefit 110

AWS Big Data

JULY 20, 2023

Moreover, running advanced analytics and ML on disparate data sources proved challenging. To overcome these issues, Orca decided to build a data lake. By decoupling storage and compute, data lakes promote cost-effective storage and processing of big data.

Data Lake 83

Data Lake Analytics Snapshot Optimization 83

AWS Big Data

MAY 30, 2023

We also demonstrate how to ingest streaming data to a transactional data lake using Apache Hudi to achieve incremental updates with ACID transactions. Solution overview For our example use case, streaming data is coming through Amazon Kinesis Data Streams , and reference data is managed in MySQL.

Data Lake 76

Data Lake Data Analytics Analytics Data Processing 76

Jet Global

MAY 24, 2022

Companies are generating more data than ever before, and it’s falling on the finance team to make sense of the meaning behind all those numbers. What can be done to increase management leverage, create more value with fewer resources, and in doing so, deliver higher value to the organization?? Data silos require a lot of extra work.

Finance 52

Finance Reporting Sales Software 52