AWS Big Data

MARCH 1, 2024

One of the bank’s key challenges related to strict cybersecurity requirements is to implement field level encryption for personally identifiable information (PII), Payment Card Industry (PCI), and data that is classified as high privacy risk (HPR). Only users with required permissions are allowed to access data in clear text.

Data Lake 101

Data Lake Data Warehouse Management Risk 101

AWS Big Data

APRIL 19, 2023

Instead, we can use automation to speed up the process of migration and reduce heavy lifting tasks, costs, and risks. We split the solution into two primary components: generating Spark job metadata and running the SQL on Amazon EMR. Generate Spark SQL metadata Our batch job consists of Hive steps scheduled to run sequentially.

Metadata 82

Metadata Testing Data Lake Consulting 82

AWS Big Data

JANUARY 22, 2024

Data as a product Treating data as a product entails three key components: the data itself, the metadata, and the associated code and infrastructure. In this approach, teams responsible for generating data are referred to as producers.

Data-driven 87

Data-driven Advertising Metadata Data Architecture 87

Domino Data Lab

APRIL 3, 2019

Data governance shows up as the fourth-most-popular kind of solution that enterprise teams were adopting or evaluating during 2019. That’s a lot of priorities – especially when you group together closely related items such as data lineage and metadata management which rank nearby. We keep feeding the monster data.

Data Governance 84

Data Governance Machine Learning Metadata Big Data 84

Domino Data Lab

AUGUST 8, 2019

I mention this here because there was a lot of overlap between current industry data governance needs and what the scientific community is working toward for scholarly infrastructure. The gist is, leveraging metadata about research datasets, projects, publications, etc., The probabilistic nature changes the risks and process required.

Data Science 71

Data Science Machine Learning Data Governance Statistics 71

AWS Big Data

APRIL 28, 2023

In this example, the analytics tool accesses the data lake on Amazon Simple Storage Service (Amazon S3) through Athena queries. As the data mesh pattern expands across domains covering more downstream services, we need a mechanism to keep IdPs and IAM role trusts continuously updated.

Data Governance 65

Data Governance Management Data-driven Data Lake 65