AWS Big Data

AUGUST 31, 2023

Amazon Redshift enables you to directly access data stored in Amazon Simple Storage Service (Amazon S3) using SQL queries and join data across your data warehouse and data lake. With Amazon Redshift, you can query the data in your S3 data lake using a central AWS Glue metastore from your Redshift data warehouse.

Data Lake 97

Data Lake Data Warehouse Metadata Data Architecture 97

AWS Big Data

SEPTEMBER 13, 2023

A modern data architecture is an evolutionary architecture pattern designed to integrate a data lake, data warehouse, and purpose-built stores with a unified governance model. The company wanted the ability to continue processing operational data in the secondary Region in the rare event of primary Region failure.

Data Lake 94

Data Lake Data Processing Metadata Snapshot 94

AWS Big Data

NOVEMBER 17, 2023

Athena provides a simplified, flexible way to analyze petabytes of data where it lives. You can analyze data or build applications from an Amazon Simple Storage Service (Amazon S3) data lake and 30 data sources, including on-premises data sources or other cloud systems using SQL or Python. Wei Zheng is a Sr.

Optimization 116

Optimization Statistics Metadata Data Lake 116

AWS Big Data

APRIL 25, 2024

In the era of data, organizations are increasingly using data lakes to store and analyze vast amounts of structured and unstructured data. Data lakes provide a centralized repository for data from various sources, enabling organizations to unlock valuable insights and drive data-driven decision-making.

Optimization 67

Optimization Data Lake Cost-Benefit Reporting 67

AWS Big Data

MAY 24, 2023

When you build your transactional data lake using Apache Iceberg to solve your functional use cases, you need to focus on operational use cases for your S3 data lake to optimize the production environment. parquet 2021-11-01 06:00:10 6.1 parquet 2021-11-01 04:33:24 6.1 availability.

Data Lake 86

Data Lake Snapshot Metadata Optimization 86

Ontotext

MARCH 8, 2023

Gartner predicts that graph technologies will be used in 80% of data and analytics innovations by 2025, up from 10% in 2021. As such, most large financial organizations have moved their data to a data lake or a data warehouse to understand and manage financial risk in one place. million users.

Enterprise 52

Enterprise Knowledge Discovery Risk Data-driven 52

AWS Big Data

FEBRUARY 1, 2023

With data volumes exhibiting a double-digit percentage growth rate year on year and the COVID pandemic disrupting global logistics in 2021, it became more critical to scale and generate near-real-time data. You can visually create, run, and monitor extract, transform, and load (ETL) pipelines to load data into your data lakes.

Optimization 52

Optimization Forecasting Data Lake Metadata 52

AWS Big Data

FEBRUARY 22, 2023

In this post, we show how Ruparupa implemented an incrementally updated data lake to get insights into their business using Amazon Simple Storage Service (Amazon S3), AWS Glue , Apache Hudi , and Amazon QuickSight. An AWS Glue ETL job, using the Apache Hudi connector, updates the S3 data lake hourly with incremental data.

Data Lake 52

Data Lake Dashboards Cost-Benefit Metadata 52

AWS Big Data

NOVEMBER 29, 2023

Amazon Redshift Serverless, generally available since 2021, allows you to run and scale analytics without having to provision and manage the data warehouse. Since GA, Redshift Serverless executed over a billion queries to power data insights for thousands of customers.

Data Warehouse 104

Data Warehouse Data Lake Analytics Machine Learning 104

CIO Business Intelligence

NOVEMBER 4, 2022

By 2020, IT had moved roughly 60% of all company data to the cloud — no minor task for an international firm that generated $16 billion in revenue in 2021, employs more than 10,000 people, and holds nine FDA- and EMA-approved drugs with an additional 30 in clinical trials. Much of Regeneron’s data, of course, is confidential.

Data Lake 122

Data Lake IT Experimentation Data-driven 122

Cloudera

NOVEMBER 17, 2022

In 2021, HBLs customers digitally carried out over 330 Mn financial transactions valued at PKR 7 Tn) in payments, a growth of 30% over 2020. We needed a solution to manage our data at scale, to provide greater experiences to our customers. HBL aims to double its banked customers by 2025. “

Management 82

Management Data Lake Consulting Unstructured Data 82

IBM Big Data Hub

APRIL 27, 2022

Gartner defines a data fabric as “a design concept that serves as an integrated layer of data and connecting processes. The data fabric architectural approach can simplify data access in an organization and facilitate self-service data consumption at scale. 11 May 2021. . 3 March 2022.

Management 83

Management Metadata Data Architecture Data Lake 83

Alation

JULY 7, 2022

Alation launched the Data Intelligence Project in the summer of 2021 to train the next generation of data leaders. With Alation, students learn the critical skills they need to curate, govern, and discover data assets in the data-driven enterprises of today. Aaron finally had the partner champions he needed.

Metadata 52

Metadata Data-driven Insurance Statistics 52

Alation

DECEMBER 14, 2021

Alation recently attended AWS re:invent 2021 … in person! Major shifts around how people use technology and data in the cloud are only just beginning. Re:Invent 2021 Keynote by AWS CEO Adam Selipsky. How do you provide access and connect the right people to the right data? What about other data sources?

Data Lake 52

Data Lake Data Warehouse Machine Learning Cost-Benefit 52

Cloudera

AUGUST 4, 2021

A top-notch system will include an easy-to-navigate data catalog that provides a single-pane view to administer and discover all data assets. The data is profiled and enhanced with rich metadata—including operational, social, and business context—creating trusted and reusable data assets and making them discoverable.

Data Governance 83

Data Governance Metadata Data-driven Enterprise 83

Cloudera

APRIL 3, 2023

The new feature this blog post is aiming to discuss about Iceberg V2 format (version 2), as the Iceberg table specification explains, the V1 format aimed to support large analytic data tables, while V2 aimed to add row level deletes and updates. CDW also simplifies administration by making multi-tenancy secure and manageable.

Data Warehouse 79

Data Warehouse Snapshot Metadata Cost-Benefit 79

Cloudera

JUNE 16, 2022

The DevOps/app dev team wants to know how data flows between such entities and understand the key performance metrics (KPMs) of these entities. For governance and security teams, the questions revolve around chain of custody, audit, metadata, access control, and lineage. Without context, streaming data is useless.”

Data Lake 84

Data Lake Manufacturing Metadata Dashboards 84

Andrew White

JANUARY 11, 2021

It was titled, The Gartner 2021 Leadership Vision for Data & Analytics Leaders. This was for the Chief Data Officer, or head of data and analytics. It is meant to be a desk-reference for that role for 2021. The fill report is here: Leadership Vision for 2021: Data and Analytics.

Data Analytics 52

Data Analytics Analytics Data-driven Finance 52

AWS Big Data

DECEMBER 21, 2023

To bring their customers the best deals and user experience, smava follows the modern data architecture principles with a data lake as a scalable, durable data store and purpose-built data stores for analytical processing and data consumption.

Data Lake 93

Data Lake Data Warehouse Data-driven B2B 93