AWS Big Data

JANUARY 24, 2024

AWS-powered data lakes, supported by the unmatched availability of Amazon Simple Storage Service (Amazon S3), can handle the scale, agility, and flexibility required to combine different data and analytics approaches. It will never remove files that are still required by a non-expired snapshot.

Snapshot 108

Snapshot Data Lake Metadata Optimization 108

AWS Big Data

NOVEMBER 17, 2023

Athena provides a simplified, flexible way to analyze petabytes of data where it lives. You can analyze data or build applications from an Amazon Simple Storage Service (Amazon S3) data lake and 30 data sources, including on-premises data sources or other cloud systems using SQL or Python.

Optimization 120

Optimization Statistics Metadata Data Lake 120

Peter James Thomas

AUGUST 18, 2018

That was the Science, here comes the Technology… A Brief Hydrology of Data Lakes. Overlapping with the above, from around 2012, I began to get involved in also designing and implementing Big Data Architectures; initially for narrow purposes and later Data Lakes spanning entire enterprises.

Data Lake 63

Data Lake Data Warehouse Data mining Statistics 63

AWS Big Data

JUNE 6, 2023

Unless, of course, the rest of their data also resides in the Google Cloud. In this post we showcase how we used AWS Glue to move siloed digital analytics data, with inconsistent arrival times, to AWS S3 (our Data Lake) and our central data warehouse (DWH), Snowflake. It consists of full-day and intraday tables.

Analytics 57

Analytics Data Lake Testing Optimization 57

In(tegrate) the Clouds

DECEMBER 21, 2015

Merv Adrian (@merv) December 19, 2015. What is the most important news item about a software company that occurred in 2015 that belongs in the capsule, and why? The resurgence of Microsoft as a cloud company was big news in 2015. Who was the biggest tech disruptor in 2015? 2016 will be the year of the data lake.

Technology 45

Technology Internet of Things Digital Transformation Software 45

AWS Big Data

FEBRUARY 1, 2023

With data volumes exhibiting a double-digit percentage growth rate year on year and the COVID pandemic disrupting global logistics in 2021, it became more critical to scale and generate near-real-time data. You can visually create, run, and monitor extract, transform, and load (ETL) pipelines to load data into your data lakes.

Optimization 52

Optimization Forecasting Data Lake Metadata 52