AWS Big Data

DECEMBER 18, 2023

Amazon S3 hosts the metadata of all the tables as a.csv file. The pipeline uses the Step Functions distributed map to read the table metadata from Amazon S3, iterate on every single item, and call the downstream AWS Glue job in parallel to export the data. The following diagram illustrates the Step Functions workflow.

Metadata 121

Metadata Visualization Data Lake Data-driven 121

AWS Big Data

MARCH 11, 2024

ORDERTOPIC" WHERE CAN_JSON_PARSE(kafka_value); The metadata column kafka_value that arrives from Amazon MSK is stored in VARBYTE format in Amazon Redshift. For this post, you use the JSON_PARSE function to convert kafka_value to a SUPER data type.

Analytics 65

Analytics Data Warehouse Optimization Metrics 65

Alation

JULY 7, 2022

In 2013 I joined American Family Insurance as a metadata analyst. I had always been fascinated by how people find, organize, and access information, so a metadata management role after school was a natural choice. The use cases for metadata are boundless, offering opportunities for innovation in every sector. The data scientist.

Metadata 52

Metadata Data-driven Insurance Statistics 52

AWS Big Data

MARCH 1, 2024

lob_risk_High_confidential_public – Contains business reports that are accessed or shared by public. lob_risk_High_confidential – Contains confidential business reports only selected principals (e.g., An AWS Glue Crawler scans the above files and catalogs metadata about them into the AWS Glue Data Catalog.

Data Lake 100

Data Lake Data Warehouse Management Risk 100

Domino Data Lab

JULY 22, 2019

I went to a meeting at Starbucks with the founder of Alation right before they launched in 2012, drawing on the proverbial back-of-the-napkin. If you look into the middle bucket, they have three things that they report in common. Now, working down to the mature part of this, they report two things in common. You know what?

Data Science 109

Data Science Machine Learning Data Governance Modeling 109

Domino Data Lab

AUGUST 8, 2019

The gist is, leveraging metadata about research datasets, projects, publications, etc., Once upon a time, circa 2012-ish, data science conferences were replete with talks about an industry hellbent on loading amazing enormous Big Data into some kind of data lake, and applying all kinds of odd astrophysics-ish approaches…for eventual PROFIT!

Data Science 71

Data Science Machine Learning Data Governance Statistics 71

O'Reilly on Data

MAY 21, 2019

Consider deep learning, a specific form of machine learning that resurfaced in 2011/2012 due to record-setting models in speech and computer vision. Metadata and artifacts needed for audits. Use ML to unlock new data types—e.g., images, audio, video. Tackle completely new use cases and applications. Source: O'Reilly.

Machine Learning 243

Machine Learning Technology Deep Learning Data Science 243

Edwin Chen

JANUARY 7, 2013

debates — Romney’s Binder (@RomneysBinder) October 17, 2012. The Storm topology attaches a spout to this Kafka queue, and the spout emits a tuple containing the query and other metadata (e.g., — Sohaib Athar (@ReallyVirtual) May 1, 2011. Or when Mitt Romney mentioned binders during the presidential debates.

Statistics 73

Statistics Machine Learning Advertising Metadata 73