AWS Big Data

NOVEMBER 22, 2023

Data lakes are designed for storing vast amounts of raw, unstructured, or semi-structured data at a low cost, and organizations share those datasets across multiple departments and teams. The queries on these large datasets read vast amounts of data and can perform complex join operations on multiple datasets.

Statistics 105

Statistics Data Lake Optimization Data-driven 105

AWS Big Data

DECEMBER 18, 2023

Solution overview One of the common functionalities involved in data pipelines is extracting data from multiple data sources and exporting it to a data lake or synchronizing the data to another database. For more information, refer to IAM Policies for invoking AWS Glue job from Step Functions.

Metadata 123

Metadata Visualization Data Lake Data-driven 123

AWS Big Data

MARCH 6, 2024

The account on the right hosts the pseudonymization service, which you can deploy using the instructions provided in the Part 1 of this series. For an overview of how to build an ACID compliant data lake using Iceberg, refer to Build a high-performance, ACID compliant, evolving data lake using Apache Iceberg on Amazon EMR.

Metrics 98

Metrics Statistics Testing Data Lake 98

AWS Big Data

DECEMBER 13, 2023

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. This solution uses Amazon Aurora MySQL hosting the example database salesdb.

Data Warehouse 102

Data Warehouse Snapshot Data Processing Management 102

AWS Big Data

NOVEMBER 16, 2023

Amazon Redshift is a popular cloud data warehouse, offering a fully managed cloud-based service that seamlessly integrates with an organization’s Amazon Simple Storage Service (Amazon S3) data lake, real-time streams, machine learning (ML) workflows, transactional workflows, and much more—all while providing up to 7.9x

Enterprise 87

Enterprise Data Warehouse Snapshot Cost-Benefit 87

AWS Big Data

FEBRUARY 16, 2023

As a QuickSight administrator, you can use AWS CloudFormation templates to migrate assets between distinct environments from development, to test, to production. For more details, refer to Amazon QuickSight resource type reference. An Amazon Redshift cluster with sample data loaded.

Data Warehouse 82

Data Warehouse Sales Visualization Data Processing 82

AWS Big Data

JUNE 13, 2023

For instructions on installing Keycloak, refer to Keycloak Downloads. Insert your specific host domain name where the Keycloak application resides in the following URL: [link] /realms/aws-realm/protocol/saml/descriptor. Vamsi Bhadriraju is a Data Architect at AWS. Sign in to your Keycloak admin dashboard.

Metadata 52

Metadata Dashboards Business Intelligence Management 52

AWS Big Data

OCTOBER 17, 2023

Tens of thousands of customers use Amazon Redshift to gain business insights from their data. With Amazon Redshift, you can use standard SQL to query data across your data warehouse, operational data stores, and data lake. For more information, refer to Configuring ongoing data replication.

Data Warehouse 89

Data Warehouse Data Processing Data Lake Management 89

Domino Data Lab

APRIL 3, 2019

“Flashpoint” (2018) – GDPR went into effect, plus major data blunders happened seemingly everywhere. Fun fact : I co-founded an e-commerce company (realistically, a mail-order catalog hosted online) in December 1992 using one of those internetworking applications called Gopher , which was vaguely popular at the time.

Data Governance 84

Data Governance Machine Learning Metadata Big Data 84

Andrew White

JANUARY 11, 2021

On January 4th I had the pleasure of hosting a webinar. It was titled, The Gartner 2021 Leadership Vision for Data & Analytics Leaders. This was for the Chief Data Officer, or head of data and analytics. It is meant to be a desk-reference for that role for 2021. Data lakes don’t offer this nor should they.

Data Analytics 52

Data Analytics Analytics Data-driven Finance 52

Cloudera

JUNE 6, 2019

The typical Cloudera Enterprise Data Hub Cluster starts with a few dozen nodes in the customer’s datacenter hosting a variety of distributed services. Over time, workloads start processing more data, tenants start onboarding more workloads, and administrators (admins) start onboarding more tenants. Cloudera Manager (CM) 6.2

Metadata 85

Metadata Data Lake Optimization Strategy 85

Cloudera

OCTOBER 7, 2021

Those decentralization efforts appeared under different monikers through time, e.g., data marts versus data warehousing implementations (a popular architectural debate in the era of structured data) then enterprise-wide data lakes versus smaller, typically BU-Specific, “data ponds”.

Metadata 122

Metadata Cost-Benefit Enterprise Interactive 122

Jet Global

FEBRUARY 23, 2024

ETL is beneficial for larger data volumes and diverse sources, and may be necessary for data architects, developers, and administrators considering factors like volume, source diversity, accuracy, and efficiency. Data Migration Data migration refers to the process of transferring data from one location or format to another.

Data Warehouse 52

Data Warehouse Reporting Data Transformation Sales 52