AWS Big Data

OCTOBER 11, 2023

Frequent materialized view refreshes on top of constantly changing base tables due to streamed data can lead to snapshot isolation errors. The second streaming data source constitutes metadata information about the call center organization and agents that gets refreshed throughout the day. We use two datasets in this post.

Management 104

Management Metadata Analytics Dashboards 104

AWS Big Data

NOVEMBER 16, 2023

Data Vault overview For a brief review of the core Data Vault premise and concepts, refer to the first post in this series. For more information, refer to Amazon Redshift database encryption. Chargeback metadata Amazon Redshift provides different pricing models to cater to different customer needs. model in Amazon Redshift.

Enterprise 83

Enterprise Data Warehouse Snapshot Cost-Benefit 83

AWS Big Data

JANUARY 27, 2023

The AWS Glue Data Catalog provides a uniform repository where disparate systems can store and find metadata to keep track of data in data silos. With unified metadata, both data processing and data consuming applications can access the tables using the same metadata. For metadata read/write, Flink has the catalog interface.

Data Lake 61

Data Lake Metadata Business Analysis Data-driven 61

AWS Big Data

AUGUST 16, 2023

Iceberg captures metadata information on the state of datasets as they evolve and change over time. AWS Glue crawlers will extract schema information and update the location of Iceberg metadata and schema updates in the Data Catalog. For more details, refer to Creating Apache Iceberg tables. Choose Create.

Data Lake 75

Data Lake Metadata Snapshot Management 75

AWS Big Data

MARCH 3, 2023

Although this post uses an Aurora PostgreSQL database hosted on AWS as the data source, the solution can be extended to ingest data from any of the AWS DMS supported databases hosted on your data centers. Let’s refer to this S3 bucket as the raw layer. Refer to Submitting EMR Serverless jobs from Airflow for additional details.

Data Lake 80

Data Lake Dashboards Metrics Metadata 80

Cloudera

JULY 13, 2022

The service provides simple, easy-to-use, and feature-rich data movement capability to deliver data and metadata where it is needed, and has secure data backup and disaster recovery functionality. In this method, you prepare the data for migration, and then set up the replication plugin to use a snapshot to migrate your data.

Snapshot 86

Snapshot Management Cost-Benefit Metadata 86