AWS Big Data

APRIL 3, 2024

Today, we are pleased to announce that Amazon DataZone is now able to present data quality information for data assets. Other organizations monitor the quality of their data through third-party solutions. Additionally, Amazon DataZone now offers APIs for importing data quality scores from external systems.

Data Quality 91

Data Quality Visualization Metadata Metrics 91

AWS Big Data

NOVEMBER 16, 2023

Redshift provisioned clusters also support query monitoring rules to define metrics-based performance boundaries for workload management queues and the action that should be taken when a query goes beyond those boundaries. A predicate consists of a metric, a comparison condition (=, ), and a value.

Enterprise 86

Enterprise Data Warehouse Snapshot Cost-Benefit 86

AWS Big Data

NOVEMBER 6, 2023

You can see the time each task spends idling while waiting for the Redshift cluster to be created, snapshotted, and paused. The following graph describes a simple data quality check pipeline using setup and teardown tasks. The Cluster Activity page gathers useful data to monitor your cluster’s live and historical metrics.

Metrics 106

Metrics Metadata Snapshot Management 106

AWS Big Data

FEBRUARY 1, 2023

We also used AWS Lambda for data processing. To further optimize and improve the developer velocity for our data consumers, we added Amazon DynamoDB as a metadata store for different data sources landing in the data lake. Clients access this data store with an API’s.

Optimization 52

Optimization Forecasting Data Lake Metadata 52

AWS Big Data

JULY 20, 2023

Prior to the creation of the data lake, Orca’s data was distributed among various data silos, each owned by a different team with its own data pipelines and technology stack. Moreover, running advanced analytics and ML on disparate data sources proved challenging.

Data Lake 88

Data Lake Analytics Snapshot Optimization 88

Alation

AUGUST 26, 2021

What Is Data Intelligence? Data intelligence is a system to deliver trustworthy, reliable data. It includes intelligence about data, or metadata. IDC coined the term, stating, “data intelligence helps organizations answer six fundamental questions about data.” Yet finding data is just the beginning.

Metadata 64

Metadata Data Governance Dashboards Software 64

DataKitchen

JANUARY 23, 2023

It allows organizations to see how data is being used, where it is coming from, its quality, and how it is being transformed. DataOps Observability includes monitoring and testing the data pipeline, data quality, data testing, and alerting. Data lineage is static and often lags by weeks or months.

Testing 130

Testing Data Governance Data Quality Data-driven 130