AWS Big Data

JANUARY 18, 2024

AWS Lake Formation and the AWS Glue Data Catalog form an integral part of a data governance solution for data lakes built on Amazon Simple Storage Service (Amazon S3) with multiple AWS analytics services integrating with them. DataZone automatically manages the permissions of your shared data in the DataZone projects.

Data Lake 86

Data Lake Metadata Data Governance Statistics 86

AWS Big Data

MARCH 7, 2024

Organizations often need to manage a high volume of data that is growing at an extraordinary rate. At the same time, they need to optimize operational costs to unlock the value of this data for timely insights and do so with a consistent performance. We think of this concept as inside-out data movement. Example Corp.

Data Lake 109

Data Lake Analytics Dashboards Metrics 109

AWS Big Data

FEBRUARY 15, 2024

Many organizations use identity providers (IdPs) to authenticate users, manage their attributes, and group memberships for secure, efficient, and centralized identity management. AWS Lake Formation makes it straightforward to centrally govern, secure, and globally share data for analytics and machine learning (ML).

Management 85

Management Data Lake Sales Data Warehouse 85

AWS Big Data

FEBRUARY 27, 2024

In the ever-evolving world of finance and lending, the need for real-time, reliable, and centralized data has become paramount. Bluestone , a leading financial institution, embarked on a transformative journey to modernize its data infrastructure and transition to a data-driven organization.

Data-driven 91

Data-driven Data Lake Data Quality Data Governance 91

AWS Big Data

APRIL 18, 2024

is a cloud-based customer relationship management (CRM) software company building artificial intelligence (AI)-powered business applications that allow businesses to connect with their customers in new and personalized ways. The data lake consumers then use Apache Presto running on Amazon EMR cluster to perform one-time queries.

Optimization 83

Optimization Data Lake Management Key Performance Indicator 83

AWS Big Data

OCTOBER 4, 2023

Amazon DataZone enables customers to discover, access, share, and govern data at scale across organizational boundaries, reducing the undifferentiated heavy lifting of making data and analytics tools accessible to everyone in the organization. This is challenging because access to data is managed differently by each of the tools.

Metadata 96

Metadata Data Lake Publishing Data Governance 96

AWS Big Data

NOVEMBER 22, 2023

Today, we’re making available a new capability of AWS Glue Data Catalog that allows generating column-level statistics for AWS Glue tables. Data lakes are designed for storing vast amounts of raw, unstructured, or semi-structured data at a low cost, and organizations share those datasets across multiple departments and teams.

Statistics 98

Statistics Data Lake Optimization Data-driven 98

AWS Big Data

NOVEMBER 22, 2023

Apache Hudi is an open table format that brings database and data warehouse capabilities to data lakes. Apache Hudi helps data engineers manage complex challenges, such as managing continuously evolving datasets with transactions while maintaining query performance.

Data Lake 89

Data Lake Snapshot Metadata Optimization 89

AWS Big Data

APRIL 26, 2024

To enable your workforce users for analytics with fine-grained data access controls and audit data access, you might have to create multiple AWS Identity and Access Management (IAM) roles with different data permissions and map the workforce users to one of those roles. We use Okta as the IdP for this demonstration.

Analytics 85

Analytics Data Lake Management Enterprise 85

AWS Big Data

NOVEMBER 29, 2023

From Talent Acquisition to Talent Management and talent insights, Eightfold offers a single AI platform that does it all. It delivers analytics and enhanced insights about the customer’s Talent Acquisition, Talent Management pipelines, and much more. Securing customer data is a top priority for Eightfold.

Metadata 100

Metadata Data Warehouse Analytics Data Analytics 100

AWS Big Data

FEBRUARY 29, 2024

Data is your generative AI differentiator, and a successful generative AI implementation depends on a robust data strategy incorporating a comprehensive data governance approach. Data governance is a critical building block across all these approaches, and we see two emerging areas of focus.

Data Governance 91

Data Governance Unstructured Data Metadata Data Lake 91

Octopai

JUNE 19, 2023

In an era where data is often referred to as the new oil, having a well-organized and easily accessible data catalog is no longer a luxury but a necessity as organizations deal with the deluge of too much data (data bloatedness) coming from every system and landscape.

Metadata 88

Metadata Data Governance Measurement Risk Management 88

CIO Business Intelligence

OCTOBER 25, 2023

For example, to ensure consistency, access control should be centrally managed, with one identification and authentication scheme to be used by all APIs. Data format should also be centrally managed to ensure uniformity. The enterprise data model must clearly indicate who is accountable for which data.

Strategy 137

Strategy Software Enterprise Technology 137

AWS Big Data

FEBRUARY 20, 2024

You can ingest and integrate data from multiple Internet of Things (IoT) sensors to get insights. However, you may have to integrate data from multiple IoT sensor devices to derive analytics like equipment health information from all the sensors based on common data elements.

IoT 90

IoT Analytics Prescriptive Analytics Internet of Things 90

Cloudera

MARCH 2, 2023

Recently, we announced enhanced multi-function analytics support in Cloudera Data Platform (CDP) with Apache Iceberg. Iceberg is a high-performance open table format for huge analytic data sets. This enables you to maximize utilization of streaming data at scale. The Catalog Type should be set to Hive.

Snapshot 113

Snapshot Data Processing Metadata Management 113

AWS Big Data

OCTOBER 23, 2023

And because Athena is serverless and fully managed, you can run your workloads without worrying about the underlying infrastructure. Before you run these workloads, most customers run SQL queries to interactively extract, filter, join, and aggregate data into a shape that can be used for decision-making, model training, or inference.

Data Lake 98

Data Lake Visualization Optimization Interactive 98

AWS Big Data

MARCH 4, 2024

As enterprises collect increasing amounts of data from various sources, the structure and organization of that data often need to change over time to meet evolving analytical needs. Schema evolution enables adding, deleting, renaming, or modifying columns without needing to rewrite existing data. Query the data using Athena.

Snapshot 108

Snapshot Data Lake Metadata Recreation/Entertainment 108

AWS Big Data

APRIL 30, 2024

Today, we’re excited to announce general availability of Amazon Q data integration in AWS Glue. Amazon Q data integration, a new generative AI-powered capability of Amazon Q Developer , enables you to build data integration pipelines using natural language.

Data Integration 99

Data Integration Data Lake Data Warehouse Software 99

AWS Big Data

JULY 14, 2023

FinAuto has a unique position to look across FinOps and provide solutions that help satisfy multiple use cases with accurate, consistent, and governed delivery of data and related services. These datasets can then be used to power front end systems, ML pipelines, and data engineering teams.

Finance 79

Finance Metadata Big Data Recreation/Entertainment 79

AWS Big Data

FEBRUARY 12, 2024

Public health organizations need access to data insights that they can quickly act upon, especially in times of health emergencies, when data needs to be updated multiple times daily. Instead, they rely on up-to-date dashboards that help them visualize data insights to make informed decisions quickly.

Publishing 87

Publishing Dashboards Metadata Visualization 87

AWS Big Data

AUGUST 3, 2023

Today’s modern data lakes span multiple accounts, AWS Regions, and lines of business in organizations. It’s important that their data solution gives them the ability to share and access data securely and safely across Regions. A resource link is a Data Catalog object that is a link to a database or table.

Data Lake 82

Data Lake Metadata Management Data Processing 82

AWS Big Data

NOVEMBER 6, 2023

Amazon EMR Studio is an integrated development environment (IDE) that makes it straightforward for data scientists and data engineers to develop, visualize, and debug data engineering and data science applications written in R, Python, Scala, and PySpark. We’re happy to introduce runtime roles for EMR Studio Workspaces.

Data Lake 90

Data Lake Sales Management Testing 90

AWS Big Data

MARCH 29, 2024

QuickSight makes it straightforward for business users to visualize data in interactive dashboards and reports. You can slice data by different dimensions like job name, see anomalies, and share reports securely across your organization. With these insights, teams have the visibility to make data integration pipelines more efficient.

Metrics 101

Metrics Visualization Dashboards Interactive 101

Jen Stirrup

MARCH 29, 2023

Despite technological advances, businesses still need help working with data. Data is complex, and companies often need to catch up on what they are trying to achieve. They are also starting to realize – and accept – that data is challenging. What does a sound, intelligent data foundation give you?

Metadata 120

Metadata Data Warehouse Data Integration Business Intelligence 120

AWS Big Data

APRIL 25, 2024

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed orchestration service for Apache Airflow that you can use to set up and operate data pipelines in the cloud at scale. By using multiple AWS accounts, organizations can effectively scale their workloads and manage their complexity as they grow.

Metadata 101

Metadata Data Processing Management Testing 101

AWS Big Data

JANUARY 22, 2024

The company uses AWS Cloud services to build data-driven products and scale engineering best practices. To ensure a sustainable data platform amid growth and profitability phases, their tech teams adopted a decentralized data mesh architecture. The solution Acast implemented is a data mesh, architected on AWS.

Data-driven 82

Data-driven Advertising Metadata Data Architecture 82

AWS Big Data

AUGUST 16, 2023

Iceberg has become very popular for its support for ACID transactions in data lakes and features like schema and partition evolution, time travel, and rollback. AWS Glue crawlers now support Iceberg tables, enabling you to use the AWS Glue Data Catalog and migrate from other Iceberg catalogs easier.

Data Lake 74

Data Lake Metadata Snapshot Management 74

Smart Data Collective

JULY 15, 2021

A data catalog serves the same purpose. By using metadata (or short descriptions), data catalogs help companies gather, organize, retrieve, and manage information. You can think of a data catalog as an enhanced Access database or library card catalog system. What Does a Data Catalog Do?

Metadata 127

Metadata IT Data-driven Data Quality 127

AWS Big Data

APRIL 3, 2024

Today, we are pleased to announce that Amazon DataZone is now able to present data quality information for data assets. Other organizations monitor the quality of their data through third-party solutions. Amazon DataZone now integrates directly with AWS Glue to display data quality scores for AWS Glue Data Catalog assets.

Data Quality 85

Data Quality Visualization Metadata Metrics 85

AWS Big Data

OCTOBER 11, 2023

Under the Transparency in Coverage (TCR) rule , hospitals and payors to publish their pricing data in a machine-readable format. The data in the machine-readable files can provide valuable insights to understand the true cost of healthcare services and compare prices and quality across hospitals.

Visualization 92

Visualization Dashboards Data-driven Gap analysis 92

CIO Business Intelligence

DECEMBER 19, 2023

IT service delivery enables an organization to give end users access to essential IT services by designing, developing, and deploying key technology resources, including applications and data. Develop a structured service catalog and adapt Achieving service delivery excellence is not a one-size-fits-all process.

IT 139

IT Testing Metrics Optimization 139

AWS Big Data

MARCH 12, 2024

In recent years, data lakes have become a mainstream architecture, and data quality validation is a critical factor to improve the reusability and consistency of the data. In this post, we provide benchmark results of running increasingly complex data quality rulesets over a predefined test dataset.

Data Quality 85

Data Quality Measurement Testing Visualization 85

AWS Big Data

SEPTEMBER 7, 2023

Customers of all sizes and industries use Amazon Simple Storage Service (Amazon S3) to store data globally for a variety of use cases. Customers want to know how their data is being accessed, when it is being accessed, and who is accessing it. With exponential growth in data volume, centralized monitoring becomes challenging.

Metadata 95

Metadata Dashboards Metrics Visualization 95

AWS Big Data

SEPTEMBER 26, 2023

AWS Lake Formation helps you centrally govern, secure, and globally share data for analytics and machine learning. With Lake Formation, you can manage access control for your data lake data in Amazon Simple Storage Service (Amazon S3 ) and its metadata in AWS Glue Data Catalog in one place with familiar database-style features.

Data Lake 65

Data Lake Metadata Management Modeling 65

Ontotext

APRIL 10, 2024

The data ecosystem today is crowded with dazzling buzzwords, all fighting for investment dollars. A survey in 2021 found that a data company was being funded every 45 minutes. Data ecosystems have become jungles and in spite of all the technology, data teams are struggling to create a modern data experience.

Metadata 52

Metadata Data Lake Data Warehouse Data Quality 52

AWS Big Data

FEBRUARY 16, 2024

Many customers are extending their data warehouse capabilities to their data lake with Amazon Redshift. They are looking to further enhance their security posture where they can enforce access policies on their data lakes based on Amazon Simple Storage Service (Amazon S3).

Data Lake 93

Data Lake Data Warehouse Testing Business Objectives 93

AWS Big Data

APRIL 4, 2024

Many enterprises are migrating their on-premises data stores to the AWS Cloud. During data migration, a key requirement is to validate all the data that has been moved from source to target. This data validation is a critical step, and if not done correctly, may result in the failure of the entire project.

Data Quality 60

Data Quality Data Lake Data Warehouse Data-driven 60