AWS Big Data

AUGUST 9, 2023

The combination of a data lake in a serverless paradigm brings significant cost and performance benefits. By monitoring application logs, you can gain insights into job execution, troubleshoot issues promptly to ensure the overall health and reliability of data pipelines.

Data Lake 90

Data Lake Metrics Testing Cost-Benefit 90

AWS Big Data

APRIL 11, 2023

With the proliferation of IoT devices and the abundance of data generated by them, it has become possible to collect real-time data on inventory levels, customer behavior, and other key metrics. In the inventory management and forecasting solution, AWS Glue is recommended for data transformation.

Forecasting 78

Forecasting Management IoT Data-driven 78

AWS Big Data

NOVEMBER 9, 2023

It does this by helping teams handle the T in ETL (extract, transform, and load) processes. It allows users to write data transformation code, run it, and test the output, all within the framework it provides. They represent an instantaneous stream of information that can be used for real-time analytics and decision-making.

Data Warehouse 110

Data Warehouse Testing Data Quality Reporting 110

AWS Big Data

FEBRUARY 6, 2023

Use case overview Migrating Hadoop workloads to Amazon EMR accelerates big data analytics modernization, increases productivity, and reduces operational cost. Refactoring coupled compute and storage to a decoupling architecture is a modern data solution. Jiseong Kim is a Senior Data Architect at AWS ProServe.

Cost-Benefit 52

Cost-Benefit Data Lake Dashboards Big Data 52

AWS Big Data

JULY 20, 2023

With data becoming the driving force behind many industries today, having a modern data architecture is pivotal for organizations to be successful. In this post, we describe Orca’s journey building a transactional data lake using Amazon Simple Storage Service (Amazon S3), Apache Iceberg, and AWS Analytics.

Data Lake 90

Data Lake Analytics Snapshot Optimization 90

DataKitchen

JULY 27, 2023

Let’s go through the ten Azure data pipeline tools Azure Data Factory : This cloud-based data integration service allows you to create data-driven workflows for orchestrating and automating data movement and transformation. You can use it for big data analytics and machine learning workloads.

Machine Learning 130

Machine Learning Cost-Benefit Data Transformation Testing 130

AWS Big Data

MARCH 3, 2023

Building data lakes from continuously changing transactional data of databases and keeping data lakes up to date is a complex task and can be an operational challenge. You can then apply transformations and store data in Delta format for managing inserts, updates, and deletes.

Data Lake 86

Data Lake Dashboards Metrics Metadata 86