AWS Big Data

MARCH 4, 2024

Apache Iceberg manages these schema changes in a backward-compatible way through its innovative metadata table evolution architecture. For instance, an ecommerce marketplace may initially partition order data by day. Lake Formation helps you centrally manage, secure, and globally share data for analytics and machine learning.

Snapshot 112

Snapshot Data Lake Metadata Recreation/Entertainment 112

AWS Big Data

NOVEMBER 17, 2023

Starting today, the Athena SQL engine uses a cost-based optimizer (CBO), a new feature that uses table and column statistics stored in the AWS Glue Data Catalog as part of the table’s metadata. Testing on the TPC-DS benchmark showed an 11% improvement in overall query performance when using CBO compared to without it.

Optimization 112

Optimization Statistics Metadata Data Lake 112

Domino Data Lab

APRIL 3, 2019

It includes perspectives about current issues, themes, vendors, and products for data governance. My interest in data governance (DG) began with the recent industry surveys by O’Reilly Media about enterprise adoption of “ABC” (AI, Big Data, Cloud). We keep feeding the monster data. the flywheel effect.

Data Governance 84

Data Governance Machine Learning Metadata Big Data 84

AWS Big Data

OCTOBER 11, 2023

It shows a call center streaming data source that sends the latest call center feed in every 15 seconds. The second streaming data source constitutes metadata information about the call center organization and agents that gets refreshed throughout the day. We use two datasets in this post.

Management 105

Management Metadata Analytics Dashboards 105

Domino Data Lab

AUGUST 8, 2019

I mention this here because there was a lot of overlap between current industry data governance needs and what the scientific community is working toward for scholarly infrastructure. The gist is, leveraging metadata about research datasets, projects, publications, etc., 2018 – Global reckoning about data governance, aka “Oops!

Data Science 71

Data Science Machine Learning Data Governance Statistics 71