Cloudera

JUNE 7, 2023

By now, almost everyone across the tech landscape has heard of the Zero Trust (ZT) security model, which assumes that every device, application, or user attempting to access a network is not to be trusted (see NIST definitions below). Understanding your data is critical to protecting the data.

Metadata 81

Metadata Data Lake Optimization Modeling 81

AWS Big Data

SEPTEMBER 13, 2023

A modern data architecture is an evolutionary architecture pattern designed to integrate a data lake, data warehouse, and purpose-built stores with a unified governance model. The company wanted the ability to continue processing operational data in the secondary Region in the rare event of primary Region failure.

Data Lake 95

Data Lake Data Processing Metadata Snapshot 95

Alation

FEBRUARY 13, 2020

These are all good questions and a logical place to start your data cataloging journey. This brief definition makes several points about data catalogs—data management, searching, data inventory, and data evaluation—but all depend on the central capability to provide a collection of metadata.

Metadata 64

Metadata Data Lake Recreation/Entertainment Big Data 64

AWS Big Data

DECEMBER 5, 2023

A combination of Amazon Redshift Spectrum and COPY commands are used to ingest the survey data stored as CSV files. For the files with unknown structures, AWS Glue crawlers are used to extract metadata and create table definitions in the Data Catalog. Outside of work she enjoys traveling and trying new cuisines.

Measurement 87

Measurement Dashboards Data Warehouse Analytics 87

AWS Big Data

AUGUST 31, 2023

Amazon Redshift enables you to directly access data stored in Amazon Simple Storage Service (Amazon S3) using SQL queries and join data across your data warehouse and data lake. With Amazon Redshift, you can query the data in your S3 data lake using a central AWS Glue metastore from your Redshift data warehouse.

Data Lake 97

Data Lake Data Warehouse Metadata Data Architecture 97

Alation

APRIL 18, 2023

For any data user in an enterprise today, data profiling is a key tool for resolving data quality issues and building new data solutions. In this blog, we’ll cover the definition of data profiling, top use cases, and share important techniques and best practices for data profiling today.

IT 52

IT Metadata Data Quality Data Governance 52

Cloudera

SEPTEMBER 15, 2022

Mark: The first element in the process is the link between the source data and the entry point into the data platform. At Ramsey International (RI), we refer to that layer in the architecture as the foundation, but others call it a staging area, raw zone, or even a source data lake. What is a data fabric?

Data Lake 81

Data Lake Data Architecture Data-driven Data Warehouse 81

erwin

OCTOBER 31, 2019

What data do we have and where is it? Data is a critical asset used to operate, manage and grow a business. While sometimes at rest in databases, data lakes and data warehouses; a large percentage is federated and integrated across the enterprise, introducing governance, manageability and risk issues that must be managed.

Data Governance 105

Data Governance Metadata Risk Data Lake 105

AWS Big Data

APRIL 25, 2024

In the era of data, organizations are increasingly using data lakes to store and analyze vast amounts of structured and unstructured data. Data lakes provide a centralized repository for data from various sources, enabling organizations to unlock valuable insights and drive data-driven decision-making.

Optimization 65

Optimization Data Lake Cost-Benefit Reporting 65

Octopai

MAY 30, 2019

At the most basic level, data catalogs help you organize your company’s massive datasets. Most enterprises have huge data lakes with millions of touchpoints all living in the dark. They have little in the way of definition or categorization. Folding In Metadata Automation. Scalability.

Metadata 40

Metadata Data Governance Data Lake IoT 40

erwin

JANUARY 11, 2019

Organizations have spent a lot of time and money trying to harmonize data across diverse platforms , including cleansing, uploading metadata, converting code, defining business glossaries, tracking data transformations and so on. If you want more control over and more value from all your data, join us for a demo of erwin MM.

Data Governance 40

Data Governance Risk Metadata Management 40

AWS Big Data

NOVEMBER 22, 2023

Apache Hudi is an open table format that brings database and data warehouse capabilities to data lakes. Apache Hudi helps data engineers manage complex challenges, such as managing continuously evolving datasets with transactions while maintaining query performance.

Data Lake 97

Data Lake Snapshot Metadata Optimization 97

AWS Big Data

JUNE 7, 2023

Cargotec captures terabytes of IoT telemetry data from their machinery operated by numerous customers across the globe. This data needs to be ingested into a data lake, transformed, and made available for analytics, machine learning (ML), and visualization. The target accounts read data from the source account S3 buckets.

Metadata 55

Metadata Data Lake Machine Learning Big Data 55

AWS Big Data

MAY 24, 2023

When you build your transactional data lake using Apache Iceberg to solve your functional use cases, you need to focus on operational use cases for your S3 data lake to optimize the production environment. availability. impl":"org.apache.iceberg.aws.s3.S3FileIO", parquet") df.sortWithinPartitions("review_date").writeTo("dev.db.amazon_reviews_iceberg").append()

Data Lake 88

Data Lake Snapshot Metadata Optimization 88

Domino Data Lab

APRIL 3, 2019

Data governance shows up as the fourth-most-popular kind of solution that enterprise teams were adopting or evaluating during 2019. That’s a lot of priorities – especially when you group together closely related items such as data lineage and metadata management which rank nearby. Definition and Descriptions.

Data Governance 84

Data Governance Machine Learning Metadata Big Data 84

Alation

JUNE 14, 2021

Indeed, automation is a key element to data catalog features, which enhance data security. Selecting a Data Catalog. To support data security, an effective data catalog should have features, like a business glossary, wiki-like articles, and metadata management. And, finding data is only half the battle.

Data Governance 97

Data Governance Recreation/Entertainment Data Lake Digital Transformation 97

AWS Big Data

OCTOBER 4, 2023

Business-driven domains – A DataZone domain represents the distinct boundary of a line of business (LOB) or a business area within an organization that can manage its own data, including its own data assets, its own definition of data or business terminology, and may have its own governing standards.

Metadata 104

Metadata Data Lake Publishing Data Governance 104

Alation

JANUARY 26, 2023

Offer the right tools Data stewardship is greatly simplified when the right tools are on hand. So ask yourself, does your steward have the software to spot issues with data quality, for example? Do they have a system to manage the metadata for given assets? One example is the EU’s General Data Protection Regulation (GDPR).

Data Governance 52

Data Governance Strategy Data Quality Marketing 52

Alation

FEBRUARY 20, 2020

When it was no longer a hard requirement that a physical data model be created upon the ingestion of data, there was a resulting drop in richness of the description and consistency of the data stored in Hadoop. You did not have to understand or prepare the data to get it into Hadoop, so people rarely did.

Data Lake 52

Data Lake Metadata Structured Data Big Data 52

AWS Big Data

JANUARY 24, 2023

This solution only replicates metadata in the Data Catalog, not the actual underlying data. To have a redundant data lake using Lake Formation and AWS Glue in an additional Region, we recommend replicating the Amazon S3-based storage using S3 replication , S3 sync, aws-s3-copy-sync-using-batch or S3 Batch replication process.

Data Architecture 70

Data Architecture Metadata Data Lake Snapshot 70

Ontotext

MARCH 8, 2023

Rich metadata and semantic modeling continue to drive the matching of 50K training materials to specific curricula, leading new, data-driven, audience-based marketing efforts that demonstrate how the recommender service is achieving increased engagement and performance from over 2.3 million users.

Enterprise 52

Enterprise Knowledge Discovery Risk Data-driven 52

AWS Big Data

MARCH 1, 2024

One of the bank’s key challenges related to strict cybersecurity requirements is to implement field level encryption for personally identifiable information (PII), Payment Card Industry (PCI), and data that is classified as high privacy risk (HPR). Only users with required permissions are allowed to access data in clear text.

Data Lake 103

Data Lake Data Warehouse Management Risk 103

IBM Big Data Hub

OCTOBER 4, 2023

A data lakehouse architecture combines the performance of data warehouses with the flexibility of data lakes, to address the challenges of today’s complex data landscape and scale AI. New insights and relationships are found in this combination. All of this supports the use of AI.

Data Lake 82

Data Lake Metadata Cost-Benefit Data Warehouse 82

Alation

JANUARY 13, 2022

Thoughtworks says data mesh is key to moving beyond a monolithic data lake. Spoiler alert: data fabric and data mesh are independent design concepts that are, in fact, quite complementary. Thoughtworks says data mesh is key to moving beyond a monolithic data lake 2. Gartner on Data Fabric.

Data Lake 59

Data Lake Metadata Data-driven Data Governance 59

Cloudera

FEBRUARY 8, 2024

Note that the materialized view definition contains the ‘stored by iceberg’ clause. The snapshotId of the source tables involved in the materialized view are also maintained in the metadata. Let’s create a materialized view that joins the three tables, has filter conditions, and does grouped aggregation.

Snapshot 90

Snapshot Metadata Cost-Benefit Data Warehouse 90

IBM Big Data Hub

JANUARY 18, 2023

In another decade, the internet and mobile started the generate data of unforeseen volume, variety and velocity. It required a different data platform solution. Hence, Data Lake emerged, which handles unstructured and structured data with huge volume. Data fabric promotes data discoverability.

Data Lake 52

Data Lake Data Warehouse Data-driven Metadata 52

erwin

DECEMBER 17, 2020

By having a single definition of something, complex ETL doesn’t have to be performed repeatedly. Once something is defined, then then everyone can map to the standard definition of what the data means. Compliance: It improves data governance to comply with such regulations as the General Data Protection Regulation (GDPR).

Modeling 90

Modeling Metadata Data-driven Data Lake 90

AWS Big Data

DECEMBER 18, 2023

Solution overview One of the common functionalities involved in data pipelines is extracting data from multiple data sources and exporting it to a data lake or synchronizing the data to another database. There are multiple tables related to customers and order data in the RDS database.

Metadata 125

Metadata Visualization Data Lake Data-driven 125

DataKitchen

OCTOBER 11, 2021

First, you must understand the existing challenges of the data team, including the data architecture and end-to-end toolchain. Second, you must establish a definition of “done.” In DataOps, the definition of done includes more than just some working code. Definition of Done. Monitoring Job Metadata.

Testing 152

Testing Metadata Dashboards Statistics 152

Alation

AUGUST 11, 2022

Today, the brightest minds in our industry are targeting the massive proliferation of data volumes and the accompanying but hard-to-find value locked within all that data. We chatted about industry trends, why decentralization has become a hot topic in the data world, and how metadata drives many data-centric use cases.

Metadata 52

Metadata Data Warehouse Data Quality Data Lake 52

Cloudera

JANUARY 21, 2021

With CDW, as an integrated service of CDP, your line of business gets immediate resources needed for faster application launches and expedited data access, all while protecting the company’s multi-year investment in centralized data management, security, and governance. Proprietary file formats mean no one else is invited in!

Data Lake 93

Data Lake Data Warehouse IT Analytics 93

Cloudera

MARCH 31, 2021

Customers who have chosen Google Cloud as their cloud platform can now use CDP Public Cloud to create secure governed data lakes in their own cloud accounts and deliver security, compliance and metadata management across multiple compute clusters. Data Preparation (Apache Spark and Apache Hive) .

Data Lake 92

Data Lake Metadata Enterprise Analytics 92

Cloudera

NOVEMBER 5, 2020

When you register an Environment in CDP, a Data Lake is automatically deployed for that environment. Data Lake security and governance is managed by a shared set of services running within a Data Lake cluster. Apache Atlas — metadata management and governance: lineage, analytics, attributes.

Data Lake 72

Data Lake Data Warehouse Metadata Optimization 72

Timo Elliott

NOVEMBER 30, 2022

Gartner says that data is a liability – after all, it costs you money to collect, and it has risks, the very definition of a liability. To turn it into an asset, you actually have to do something with the data, to change something in the way you do business. And that’s what often goes wrong.

Recreation/Entertainment 98

Recreation/Entertainment Metadata Data Warehouse Cost-Benefit 98

AWS Big Data

OCTOBER 2, 2023

JSON data in Amazon Redshift Amazon Redshift enables storage, processing, and analytics on JSON data through the SUPER data type, PartiQL language, materialized views, and data lake queries. This means that when a message is serialized, it includes both the schema definition and the data values.

Cost-Benefit 90

Cost-Benefit Metadata Structured Data Management 90

AWS Big Data

JUNE 12, 2023

This job extracts data from the Kafka topics, deserializes it using the schema information from the Data Catalog table, and loads it into Amazon S3. It’s important to note that the schema in the Data Catalog table serves as the source of truth for the AWS Glue streaming job.

Management 83

Management Metadata Testing Internet of Things 83

Alation

APRIL 14, 2022

In today’s AI/ML-driven world of data analytics, explainability needs a repository just as much as those doing the explaining need access to metadata, EG, information about the data being used. The Cloud Data Migration Challenge. Pushing data to a data lake and assuming it is ready for use is shortsighted.

Metadata 59

Metadata Data Governance Modeling Data-driven 59