AWS Big Data

MARCH 10, 2023

They enable transactions on top of data lakes and can simplify data storage, management, ingestion, and processing. These transactional data lakes combine features from both the data lake and the data warehouse. One important aspect to a successful data strategy for any organization is data governance.

Data Lake 102

Data Lake Sales Data Warehouse Snapshot 102

AWS Big Data

APRIL 3, 2024

If the asset has AWS Glue Data Quality enabled, you can now quickly visualize the data quality score directly in the catalog search pane. By selecting the corresponding asset, you can understand its content through the readme, glossary terms , and technical and business metadata. You can download the complete PySpark script.

Data Quality 89

Data Quality Visualization Metadata Metrics 89

AWS Big Data

SEPTEMBER 13, 2023

The File Manager Lambda function consumes those messages, parses the metadata, and inserts the metadata to the DynamoDB table odpf_file_tracker. We use the following terminology when discussing File Processor: Refresh cadence – This represents the data ingestion frequency (for example, 10 minutes).

Data Lake 91

Data Lake Data Processing Metadata Snapshot 91