AWS Big Data

OCTOBER 31, 2023

This blog post is co-written with James Sun from Snowflake. Customers rely on data from different sources such as mobile applications, clickstream events from websites, historical data, and more to deduce meaningful patterns to optimize their products, services, and processes.

Data Processing 93

Data Processing Management Publishing Testing 93

AWS Big Data

APRIL 4, 2024

Many enterprises are migrating their on-premises data stores to the AWS Cloud. During data migration, a key requirement is to validate all the data that has been moved from source to target. This data validation is a critical step, and if not done correctly, may result in the failure of the entire project.

Data Quality 65

Data Quality Data Lake Data Warehouse Data-driven 65

AWS Big Data

JANUARY 8, 2024

We’re living in the age of real-time data and insights, driven by low-latency data streaming applications. The volume of time-sensitive data produced is increasing rapidly, with different formats of data being introduced across new businesses and customer use cases.

Analytics 116

Analytics IoT Data-driven Snapshot 116

IBM Big Data Hub

FEBRUARY 29, 2024

More than 110 countries at the United Nations’ COP28 climate change conference agreed to triple that capacity by 2030, and global investment in clean energy transition hit a record high of USD 1.8 But with all of this new capacity, how are renewable energy resources really being used? trillion in 2023.

IoT 102

IoT Internet of Things Optimization Manufacturing 102

AWS Big Data

JULY 21, 2023

Data-driven organizations treat data as an asset and use it across different lines of business (LOBs) to drive timely insights and better business decisions. This leads to having data across many instances of data warehouses and data lakes using a modern data architecture in separate AWS accounts.

Data Lake 88

Data Lake Data Warehouse Marketing Management 88

AWS Big Data

NOVEMBER 29, 2023

dbt is an open source, SQL-first templating engine that allows you to write repeatable and extensible data transforms in Python and SQL. dbt is predominantly used by data warehouses (such as Amazon Redshift ) customers who are looking to keep their data transform logic separate from storage and engine.

Data Lake 107

Data Lake Management Metrics Data Warehouse 107

AWS Big Data

AUGUST 7, 2023

Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud. Tens of thousands of customers use Amazon Redshift to process exabytes of data every day to power their analytics workloads. You pay only the associated Forecast costs. View Amazon Forecast pricing for details.

Forecasting 86

Forecasting Data Warehouse Machine Learning Modeling 86

AWS Big Data

JUNE 2, 2023

Amazon Redshift is a widely used, fully managed, petabyte-scale cloud data warehouse. Tens of thousands of customers use Amazon Redshift to process exabytes of data every day to power their analytic workloads. This pre-built solution scales to load data in parallel using input parameters.

Metadata 114

Metadata Data Warehouse Big Data Testing 114

AWS Big Data

APRIL 3, 2023

Tens of thousands of customers run business-critical workloads on Amazon Redshift , AWS’s fast, petabyte-scale cloud data warehouse delivering the best price-performance. With Amazon Redshift, you can query data across your data warehouse, operational data stores, and data lake using standard SQL.

Data Warehouse 100

Data Warehouse Testing Data Lake Data-driven 100

datapine

SEPTEMBER 29, 2022

1) What Is Data Quality Management? 4) Data Quality Best Practices. 5) How Do You Measure Data Quality? 6) Data Quality Metrics Examples. 7) Data Quality Control: Use Case. 8) The Consequences Of Bad Data Quality. 9) 3 Sources Of Low-Quality Data. 10) Data Quality Solutions: Key Attributes.

Data Quality 241

Data Quality Metrics Data-driven Management 241

AWS Big Data

NOVEMBER 22, 2023

Today, we’re making available a new capability of AWS Glue Data Catalog that allows generating column-level statistics for AWS Glue tables. Data lakes are designed for storing vast amounts of raw, unstructured, or semi-structured data at a low cost, and organizations share those datasets across multiple departments and teams.

Statistics 105

Statistics Data Lake Optimization Data-driven 105

AWS Big Data

FEBRUARY 9, 2023

Customers across industries seek meaningful insights from the data captured in their Customer Relationship Management (CRM) systems. To achieve this, they combine their CRM data with a wealth of information already available in their data warehouse, enterprise systems, or other software as a service (SaaS) applications.

Data Warehouse 79

Data Warehouse Data-driven Snapshot Testing 79

AWS Big Data

JUNE 29, 2023

Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud that delivers powerful and secure insights on all your data with the best price-performance. With Amazon Redshift, you can analyze your data to derive holistic insights about your business and your customers.

Data Warehouse 61

Data Warehouse Dashboards Testing Visualization 61

DataKitchen

NOVEMBER 4, 2021

Data organizations often have a mix of centralized and decentralized activity. DataOps concerns itself with the complex flow of data across teams, data centers and organizational boundaries. It expands beyond tools and data architecture and views the data organization from the perspective of its processes and workflows.

Data Processing 246

Data Processing Data Lake Cost-Benefit Testing 246

Sisense

APRIL 17, 2020

We live in a world of data: there’s more of it than ever before, in a ceaselessly expanding array of forms and locations. Dealing with Data is your window into the ways Data Teams are tackling the challenges of this new world to help their companies and their customers thrive. What is a materialized view, anyway?

Data Warehouse 90

Data Warehouse Visualization Statistics Machine Learning 90

AWS Big Data

NOVEMBER 22, 2023

Apache Hudi is an open table format that brings database and data warehouse capabilities to data lakes. Apache Hudi helps data engineers manage complex challenges, such as managing continuously evolving datasets with transactions while maintaining query performance.

Data Lake 97

Data Lake Snapshot Metadata Optimization 97

AWS Big Data

OCTOBER 20, 2023

Today, we are pleased to announce new AWS Glue connectors for Azure Blob Storage and Azure Data Lake Storage that allow you to move data bi-directionally between Azure Blob Storage, Azure Data Lake Storage, and Amazon Simple Storage Service (Amazon S3).

Data Lake 87

Data Lake Big Data Consulting Data Warehouse 87

Sisense

JULY 8, 2020

We live in a world of data: there’s more of it than ever before, in a ceaselessly expanding array of forms and locations. Dealing with Data is your window into the ways Data Teams are tackling the challenges of this new world to help their companies and their customers thrive.

Data Warehouse 90

Data Warehouse Data Lake Data-driven Cost-Benefit 90

AWS Big Data

JANUARY 17, 2024

This allows you to simplify security and governance over transactional data lakes by providing access controls at table-, column-, and row-level permissions with your Apache Spark jobs. Many large enterprise companies seek to use their transactional data lake to gain insights and improve decision-making.

Data Lake 97

Data Lake Snapshot Big Data Data-driven 97

AWS Big Data

APRIL 5, 2023

Amazon Redshift is a massively parallel processing (MPP), fully managed petabyte-scale data warehouse that makes it simple and cost-effective to analyze all your data using existing business intelligence tools. Solution overview The following diagram describes the solution architecture.

Data Warehouse 79

Data Warehouse Management Business Intelligence Testing 79

AWS Big Data

APRIL 20, 2023

Customers use Amazon Redshift to run their business-critical analytics on petabytes of structured and semi-structured data. Apache Spark is a popular framework that you can use to build applications for use cases such as ETL (extract, transform, and load), interactive analytics, and machine learning (ML).

Data Lake 79

Data Lake Data Warehouse Sales Data-driven 79

AWS Big Data

MARCH 2, 2023

Apache Iceberg is an open table format for very large analytic datasets, which captures metadata information on the state of datasets as they evolve and change over time. It adds tables to compute engines including Spark, Trino, PrestoDB, Flink, and Hive using a high-performance table format that works just like a SQL table.

Data Lake 121

Data Lake Data Processing Metadata Snapshot 121

AWS Big Data

APRIL 20, 2023

In a modern data architecture, unified analytics enable you to access the data you need, whether it’s stored in a data lake or a data warehouse. AWS Glue provides an extensible architecture that enables users with different data processing use cases, and works well with Amazon Redshift.

Visualization 80

Visualization Data Warehouse Big Data Data Lake 80

AWS Big Data

JULY 12, 2023

This blog post is co-written with Zygimantas Koncius from Etleap. Organizations use their data to extract valuable insights and drive informed business decisions. The result is robust and flexible data products with high scalability and best-in-class query performance.

Data Warehouse 52

Data Warehouse Modeling Dashboards Data Lake 52

AWS Big Data

AUGUST 1, 2023

In the world of software engineering and development, organizations use project management tools like Atlassian Jira Cloud. Companies often take a data lake approach to their analytics, bringing data from many different systems into one place to simplify how the analytics are done.

Data Lake 87

Data Lake Data Transformation Cost-Benefit Data-driven 87

AWS Big Data

MAY 24, 2023

When you build your transactional data lake using Apache Iceberg to solve your functional use cases, you need to focus on operational use cases for your S3 data lake to optimize the production environment. Amazon S3 deletes expired objects on your behalf. With the s3.delete.tags

Data Lake 86

Data Lake Snapshot Metadata Optimization 86

CIO Business Intelligence

SEPTEMBER 8, 2022

The headlines read “Artificial Intelligence (AI) will completely transform your business.” This creates a flywheel we call The Data Network Effect, where you acquire more data, which helps create better algorithms, which drives better engagement, ultimately leading to happier customers, which then generates more data, and so on, and so on.

Machine Learning 99

Machine Learning Cost-Benefit Data-driven Digital Transformation 99

AWS Big Data

MAY 25, 2023

Data transformation plays a pivotal role in providing the necessary data insights for businesses in any organization, small and large. Picture a scenario where you, the VP of Data and Analytics, are in charge of your data and analytics environments and workloads running on AWS where you manage a team of data engineers and analysts.

Reporting 71

Reporting Metrics Optimization Data Lake 71

AWS Big Data

JUNE 6, 2023

Data is the lifeblood of modern businesses. In today’s data-driven world, companies rely on data to make informed decisions, gain a competitive edge, and provide exceptional customer experiences. However, not all data is created equal. AWS Glue Data Quality measures and monitors the quality of your dataset.

Data Quality 70

Data Quality Data Lake Visualization Data-driven 70

AWS Big Data

JULY 19, 2023

Amazon Redshift is a fully managed and petabyte-scale cloud data warehouse that is used by tens of thousands of customers to process exabytes of data every day to power their analytics workload. Overall, the post will give you a clear understanding of how to use dimensional modeling in Amazon Redshift.

Modeling 71

Modeling Sales Data Warehouse Snapshot 71

AWS Big Data

SEPTEMBER 13, 2023

The Analytics specialty practice of AWS Professional Services (AWS ProServe) helps customers across the globe with modern data architecture implementations on the AWS Cloud. In AWS ProServe-led customer engagements, the use cases we work on usually come with technical complexity and scalability requirements.

Data Lake 94

Data Lake Data Processing Metadata Snapshot 94

AWS Big Data

JULY 31, 2023

Data has become an integral part of most companies, and the complexity of data processing is increasing rapidly with the exponential growth in the amount and variety of data. AWS Glue is a serverless data integration and ETL service with the ability to scale on demand.

Sales 52

Sales Data Warehouse Visualization Testing 52

AWS Big Data

JUNE 23, 2023

This blog post is co-written with Ori Nakar from Imperva. Events and many other security data types are stored in Imperva’s Threat Research Multi-Region data lake. Imperva harnesses data to improve their business outcomes. As part of their solution, they are using Amazon QuickSight to unlock insights from their data.

Data Lake 71

Data Lake Cost-Benefit Dashboards Data Warehouse 71

CDW Research Hub

DECEMBER 4, 2019

There are many benefits of using a cloud-based data warehouse, and the market for cloud-based data warehouses is growing as organizations realize the value of making the switch from an on-premises data warehouse. The top 4 benefits of using Snowflake with Microsoft Azure.

Data Warehouse 52

Data Warehouse Data mining Data Lake Dashboards 52

AWS Big Data

MARCH 3, 2023

Building data lakes from continuously changing transactional data of databases and keeping data lakes up to date is a complex task and can be an operational challenge. A solution to this problem is to use AWS Database Migration Service (AWS DMS) for migrating historical and real-time transactional data into the data lake.

Data Lake 85

Data Lake Dashboards Metrics Metadata 85

AWS Big Data

FEBRUARY 8, 2023

Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud. Amazon Redshift enables you to run complex SQL analytics at scale and performance on terabytes to petabytes of structured and unstructured data, and make the insights widely available through popular business intelligence (BI) and analytics tools.

Data Quality 78

Data Quality Testing Data Warehouse Unstructured Data 78

AWS Big Data

MARCH 28, 2023

In a data warehouse, a dimension is a structure that categorizes facts and measures in order to enable users to answer business questions. A slowly changing dimension (SCD) is a data warehousing concept that contains relatively static data that can change slowly over a period of time.

Data Lake 73

Data Lake Testing Snapshot Sales 73