AWS Big Data

MARCH 4, 2024

As enterprises collect increasing amounts of data from various sources, the structure and organization of that data often need to change over time to meet evolving analytical needs. Schema evolution enables adding, deleting, renaming, or modifying columns without needing to rewrite existing data. Query the data using Athena.

Snapshot 110

Snapshot Data Lake Metadata Recreation/Entertainment 110

Cloudera

MARCH 2, 2023

Recently, we announced enhanced multi-function analytics support in Cloudera Data Platform (CDP) with Apache Iceberg. Iceberg is a high-performance open table format for huge analytic data sets. This enables you to maximize utilization of streaming data at scale. The Catalog Type should be set to Hive.

Snapshot 112

Snapshot Data Processing Metadata Management 112

AWS Big Data

FEBRUARY 12, 2024

Public health organizations need access to data insights that they can quickly act upon, especially in times of health emergencies, when data needs to be updated multiple times daily. Instead, they rely on up-to-date dashboards that help them visualize data insights to make informed decisions quickly.

Publishing 89

Publishing Dashboards Metadata Visualization 89

AWS Big Data

AUGUST 3, 2023

Today’s modern data lakes span multiple accounts, AWS Regions, and lines of business in organizations. It’s important that their data solution gives them the ability to share and access data securely and safely across Regions. A resource link is a Data Catalog object that is a link to a database or table.

Data Lake 84

Data Lake Metadata Management Data Processing 84

AWS Big Data

MARCH 29, 2024

QuickSight makes it straightforward for business users to visualize data in interactive dashboards and reports. You can slice data by different dimensions like job name, see anomalies, and share reports securely across your organization. With these insights, teams have the visibility to make data integration pipelines more efficient.

Metrics 103

Metrics Visualization Dashboards Interactive 103

AWS Big Data

AUGUST 16, 2023

Iceberg has become very popular for its support for ACID transactions in data lakes and features like schema and partition evolution, time travel, and rollback. AWS Glue crawlers now support Iceberg tables, enabling you to use the AWS Glue Data Catalog and migrate from other Iceberg catalogs easier.

Data Lake 76

Data Lake Metadata Snapshot Management 76

AWS Big Data

APRIL 3, 2024

Today, we are pleased to announce that Amazon DataZone is now able to present data quality information for data assets. Other organizations monitor the quality of their data through third-party solutions. Amazon DataZone now integrates directly with AWS Glue to display data quality scores for AWS Glue Data Catalog assets.

Data Quality 87

Data Quality Visualization Metadata Metrics 87

Smart Data Collective

JULY 15, 2021

A data catalog serves the same purpose. By using metadata (or short descriptions), data catalogs help companies gather, organize, retrieve, and manage information. You can think of a data catalog as an enhanced Access database or library card catalog system. What Does a Data Catalog Do?

Metadata 127

Metadata IT Data-driven Data Quality 127

AWS Big Data

MARCH 12, 2024

In recent years, data lakes have become a mainstream architecture, and data quality validation is a critical factor to improve the reusability and consistency of the data. In this post, we provide benchmark results of running increasingly complex data quality rulesets over a predefined test dataset.

Data Quality 87

Data Quality Measurement Testing Visualization 87

Ontotext

APRIL 10, 2024

The data ecosystem today is crowded with dazzling buzzwords, all fighting for investment dollars. A survey in 2021 found that a data company was being funded every 45 minutes. Data ecosystems have become jungles and in spite of all the technology, data teams are struggling to create a modern data experience.

Metadata 52

Metadata Data Lake Data Warehouse Data Quality 52

AWS Big Data

APRIL 4, 2024

Many enterprises are migrating their on-premises data stores to the AWS Cloud. During data migration, a key requirement is to validate all the data that has been moved from source to target. This data validation is a critical step, and if not done correctly, may result in the failure of the entire project.

Data Quality 61

Data Quality Data Lake Data Warehouse Data-driven 61

AWS Big Data

FEBRUARY 16, 2024

Many customers are extending their data warehouse capabilities to their data lake with Amazon Redshift. They are looking to further enhance their security posture where they can enforce access policies on their data lakes based on Amazon Simple Storage Service (Amazon S3).

Data Lake 95

Data Lake Data Warehouse Testing Business Objectives 95

CIO Business Intelligence

DECEMBER 19, 2023

IT service delivery enables an organization to give end users access to essential IT services by designing, developing, and deploying key technology resources, including applications and data. Develop a structured service catalog and adapt Achieving service delivery excellence is not a one-size-fits-all process.

IT 139

IT Testing Metrics Optimization 139

AWS Big Data

JANUARY 31, 2023

Data governance is the collection of policies, processes, and systems that organizations use to ensure the quality and appropriate handling of their data throughout its lifecycle for the purpose of generating business value.

Data Lake 80

Data Lake Data Governance Data Architecture Machine Learning 80

AWS Big Data

JUNE 15, 2023

In today’s world, customers manage vast amounts of data in their Amazon Simple Storage Service (Amazon S3) data lakes, which requires convoluted data pipelines to continuously understand the changes in the data layout and make them available to consuming systems. Choose Next.

Data Lake 86

Data Lake Metadata Cost-Benefit Management 86

Octopai

APRIL 24, 2023

This new native integration enhances our data lineage solution by providing seamless integration with one of the most powerful cloud-based data warehouses, benefiting data teams and enabling support for a broader range of data lineage, discovery, and catalog.

Cost-Benefit 96

Cost-Benefit Data-driven Data Warehouse Data Governance 96

AWS Big Data

NOVEMBER 17, 2023

Amazon DynamoDB is a fully managed, serverless, key-value NoSQL database designed to run high-performance applications at any scale. DynamoDB offers built-in security, continuous backups, automated multi-Region replication, in-memory caching, and data import and export tools.

Visualization 81

Visualization Metadata Testing Internet of Things 81

erwin

JUNE 26, 2020

If you’re serious about a data-driven strategy , you’re going to need a data catalog. Organizations need a data catalog because it enables them to create a seamless way for employees to access and consume data and business assets in an organized manner. Three Types of Metadata in a Data Catalog.

Metadata 132

Metadata Cost-Benefit Measurement Data-driven 132

CIO Business Intelligence

MARCH 24, 2023

Data governance definition Data governance is a system for defining who within an organization has authority and control over data assets and how those data assets may be used. It encompasses the people, processes, and technologies required to manage and protect data assets.

Data Governance 142

Data Governance Management Metadata Data Quality 142

AWS Big Data

NOVEMBER 10, 2023

Apache Spark is a widely-used open source distributed processing system renowned for handling large-scale data workloads. Amazon Redshift offers seamless integration with Apache Spark, allowing you to easily access your Redshift data on both Amazon Redshift provisioned clusters and Amazon Redshift Serverless.

Data Processing 91

Data Processing Data Lake Data Warehouse Optimization 91

AWS Big Data

APRIL 25, 2024

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed orchestration service for Apache Airflow that you can use to set up and operate data pipelines in the cloud at scale. By using multiple AWS accounts, organizations can effectively scale their workloads and manage their complexity as they grow.

Metadata 72

Metadata Data Processing Management Testing 72

AWS Big Data

JULY 20, 2023

Apache Hive is a SQL-based data warehouse system for processing highly distributed datasets on the Apache Hadoop platform. The Hive metastore is a repository of metadata about the SQL tables, such as database names, table names, schema, serialization and deserialization information, data location, and partition details of each table.

Data Lake 78

Data Lake Metadata Data Processing Big Data 78

AWS Big Data

MARCH 26, 2024

We are delighted to announce the release of the Data Analytics Lens. The lens consists of a lens whitepaper and an AWS-created lens available in the Lens Catalog of the AWS Well-Architected Tool. What’s new in the Data Analytics Lens? For more information on AWS Well-Architected Lenses, refer to AWS Well-Architected.

Data Analytics 55

Data Analytics Analytics Big Data Data Lake 55

CIO Business Intelligence

NOVEMBER 21, 2023

But for a select few, the deeper challenges of departmental technologies being funded, procured, and managed without IT involvement are the missed opportunities to better engage and fulfill departmental technology needs. That’s where an IT strategy that frames shadow IT as an opportunity for improved collaboration can have a profound impact.

IT 135

IT Risk Cost-Benefit Risk Management 135

AWS Big Data

SEPTEMBER 13, 2023

Streaming data has become an indispensable resource for organizations worldwide because it offers real-time insights that are crucial for data analytics. The escalating velocity and magnitude of collected data has created a demand for real-time analytics.

Management 89

Management Data Processing Interactive Metadata 89

Alation

JULY 26, 2021

How do you initiate change within a system containing many thousands of people and millions of bytes of data? During my time as a data specialist at American Family Insurance, it became clear that we had to move away from the way things had been done in the past. So you can probably imagine: The company manages a lot of data.

Insurance 98

Insurance Digital Transformation Enterprise Data Governance 98

Alation

JULY 19, 2021

Metadata is information about data. A clothing catalog or dictionary are both examples of metadata repositories. Indeed, a popular online catalog, like Amazon, offers rich metadata around products to guide shoppers: ratings, reviews, and product details are all examples of metadata. What Is Metadata? Why Is Metadata Important?

Metadata 105

Metadata Management Data Governance Machine Learning 105

AWS Big Data

MARCH 19, 2024

Companies may find themselves challenged to manage the limited pool of IP addresses. For data engineering workloads when AWS Glue is used in such a constrained network configuration, your team may sometimes face hurdles running many jobs simultaneously. These ENIs are short-lived and active until job is complete.

Optimization 86

Optimization Data-driven Management Testing 86

IBM Big Data Hub

FEBRUARY 12, 2024

Recognizing the need to harness real-time data, businesses are increasingly turning to event-driven architecture (EDA) as a strategic approach to stay ahead of the curve. This trend grows stronger as organizations realize the benefits that come from the power of real-time data streaming. Do you remember playing in the sandbox as a kid?

Data-driven 86

Data-driven Cost-Benefit Uncertainty Technology 86

AWS Big Data

JUNE 6, 2023

We are excited to announce the General Availability of AWS Glue Data Quality. Our journey started by working backward from our customers who create, manage, and operate data lakes and data warehouses for analytics and machine learning. It takes days for data engineers to identify and implement data quality rules.

Data Quality 93

Data Quality Statistics Data Lake Visualization 93

AWS Big Data

SEPTEMBER 26, 2023

This is a guest post co-written with Tyler Middleton, Experian Senior Partner Marketing Manager, and Jay Rakhe, Experian Group Product Manager. As the data privacy landscape continues to evolve, companies are increasingly seeking ways to collect and manage data while protecting privacy and intellectual property.

Advertising 76

Advertising Data-driven Marketing Interactive 76

Timo Elliott

JANUARY 6, 2022

Like the proverbial man looking for his keys under the streetlight , when it comes to enterprise data, if you only look at where the light is already shining, you can end up missing a lot. Remember that dark data is the data you have but don’t understand. So how do you find your dark data? Create a catalog.

IT 133

IT Metadata Data-driven Data Governance 133

Alation

JULY 22, 2021

Our platform combines data insights with human intelligence in pursuit of this mission. Susannah Barnes, an Alation customer and senior data governance specialist at American Family Insurance, introduced our team to faculty at the School of Information Studies of the University of Wisconsin, Milwaukee (UWM-SOIS), her alma mater.

Informatics 98

Informatics Big Data Insurance Metadata 98

Octopai

JUNE 27, 2022

This data catalog is helping boost our bottom line.”. Your data team spends more time doing high-level analysis than it does searching for relevant datasets. Your most technologically-challenged business user searches the data catalog at least once a week – without coming to you for help! But don’t worry.

Management 72

Management Metadata Visualization Data Governance 72

Ontotext

MAY 2, 2024

Data leaders today are faced with an almost impossible challenge. They are expected to understand the entire data landscape and generate business-moving insights while facing the voracious needs of different teams and the constraints of technology architecture and compliance.

Metadata 52

Metadata Data Lake Data-driven Enterprise 52

erwin

OCTOBER 8, 2020

So if you’re going to move from your data from on-premise legacy data stores and warehouse systems to the cloud, you should do it right the first time. And as you make this transition, you need to understand what data you have, know where it is located, and govern it along the way. Then you must bulk load the legacy data.

Data Governance 145

Data Governance Metadata Testing Data Lake 145

AWS Big Data

JUNE 6, 2023

AWS Glue Data Quality allows you to measure and monitor the quality of data in your data repositories. It’s important for business users to be able to see quality scores and metrics to make confident business decisions and debug data quality issues. An AWS Glue crawler crawls the results.

Data Quality 70

Data Quality Metrics Visualization Dashboards 70