AWS Big Data

MARCH 10, 2023

They enable transactions on top of data lakes and can simplify data storage, management, ingestion, and processing. These transactional data lakes combine features from both the data lake and the data warehouse. One important aspect to a successful data strategy for any organization is data governance.

Data Lake 107

Data Lake Sales Data Warehouse Snapshot 107

AWS Big Data

APRIL 3, 2024

If the asset has AWS Glue Data Quality enabled, you can now quickly visualize the data quality score directly in the catalog search pane. By selecting the corresponding asset, you can understand its content through the readme, glossary terms , and technical and business metadata.

Data Quality 93

Data Quality Visualization Metadata Metrics 93

AWS Big Data

JULY 28, 2023

By creating visual representations of data flows, organizations can gain a clear understanding of the lifecycle of personal data and identify potential vulnerabilities or compliance gaps. Note that putting a comprehensive data strategy in place is not in scope for this post. However, this is beyond the scope of this post.

Snapshot 85

Snapshot Metadata Measurement Data Warehouse 85

AWS Big Data

FEBRUARY 1, 2023

We also used AWS Lambda for data processing. To further optimize and improve the developer velocity for our data consumers, we added Amazon DynamoDB as a metadata store for different data sources landing in the data lake. Clients access this data store with an API’s.

Optimization 52

Optimization Forecasting Data Lake Metadata 52

AWS Big Data

SEPTEMBER 13, 2023

The File Manager Lambda function consumes those messages, parses the metadata, and inserts the metadata to the DynamoDB table odpf_file_tracker. We use the following terminology when discussing File Processor: Refresh cadence – This represents the data ingestion frequency (for example, 10 minutes).

Data Lake 94

Data Lake Data Processing Metadata Snapshot 94

Alation

AUGUST 26, 2021

What Is Data Intelligence? Data intelligence is a system to deliver trustworthy, reliable data. It includes intelligence about data, or metadata. IDC coined the term, stating, “data intelligence helps organizations answer six fundamental questions about data.” Yet finding data is just the beginning.

Metadata 64

Metadata Data Governance Dashboards Software 64