AWS Big Data

JANUARY 31, 2023

With OpenSearch Serverless, you get the same interactive millisecond response times as OpenSearch Service with the simplicity of a serverless environment. With OpenSearch domains, you get dedicated, secure clusters configured and optimized for your workloads in minutes. In between, you configure filters that can transform your data.

Data Processing 81

Data Processing Optimization Software Analytics 81

AWS Big Data

APRIL 3, 2023

Tens of thousands of customers run business-critical workloads on Amazon Redshift , AWS’s fast, petabyte-scale cloud data warehouse delivering the best price-performance. With Amazon Redshift, you can query data across your data warehouse, operational data stores, and data lake using standard SQL.

Data Warehouse 96

Data Warehouse Testing Data Lake Data-driven 96

AWS Big Data

JULY 6, 2023

Extracting time series on given geographical coordinates from satellite or Numerical Weather Prediction data can be challenging because of the volume of data and of its multidimensional nature (time, latitude, longitude, height, multiple parameters). It has not been specifically designed for heavy data transformation tasks.

Machine Learning 128

Machine Learning Visualization IoT Digital Transformation 128

AWS Big Data

JUNE 6, 2023

Alerts and notifications play a crucial role in maintaining data quality because they facilitate prompt and efficient responses to any data quality issues that may arise within a dataset. It simplifies your experience of monitoring and evaluating the quality of your data.

Data Quality 73

Data Quality Metrics Data-driven Visualization 73

AWS Big Data

JULY 12, 2023

Organizations use their data to extract valuable insights and drive informed business decisions. In this post, we explain how data teams can quickly configure low-latency data pipelines that ingest and model data from a variety of sources, using Etleap’s end-to-end pipelines with Amazon Redshift and dbt.

Data Warehouse 52

Data Warehouse Modeling Dashboards Data Lake 52

AWS Big Data

SEPTEMBER 13, 2023

The Analytics specialty practice of AWS Professional Services (AWS ProServe) helps customers across the globe with modern data architecture implementations on the AWS Cloud. In this post, we discuss a common use case in relation to operational data processing and the solution we built using Apache Hudi and AWS Glue.

Data Lake 89

Data Lake Data Processing Metadata Snapshot 89

AWS Big Data

MARCH 25, 2024

FMs are multimodal; they work with different data types such as text, video, audio, and images. Large language models (LLMs) are a type of FM and are pre-trained on vast amounts of text data and typically have application uses such as text generation, intelligent chatbots, or summarization.

Data Lake 88

Data Lake Unstructured Data Management Modeling 88

AWS Big Data

OCTOBER 12, 2023

You can run analytics workloads at any scale with automatic scaling that resizes resources in seconds to meet changing data volumes and processing requirements. The integration between AWS Step Functions and Amazon EMR Serverless makes it easier to manage and orchestrate big data workflows. Now, with the support for “Run a Job (.sync)”

Big Data 105

Big Data Data-driven Management Visualization 105

AWS Big Data

NOVEMBER 16, 2023

Amazon Redshift is a popular cloud data warehouse, offering a fully managed cloud-based service that seamlessly integrates with an organization’s Amazon Simple Storage Service (Amazon S3) data lake, real-time streams, machine learning (ML) workflows, transactional workflows, and much more—all while providing up to 7.9x

Enterprise 83

Enterprise Data Warehouse Data Lake Optimization 83

AWS Big Data

NOVEMBER 16, 2023

Amazon Redshift is a popular cloud data warehouse, offering a fully managed cloud-based service that seamlessly integrates with an organization’s Amazon Simple Storage Service (Amazon S3) data lake, real-time streams, machine learning (ML) workflows, transactional workflows, and much more—all while providing up to 7.9x

Enterprise 82

Enterprise Data Warehouse Snapshot Cost-Benefit 82

AWS Big Data

MAY 9, 2023

AWS Glue Studio is a graphical interface that makes it easy to create, run, and monitor extract, transform, and load (ETL) jobs in AWS Glue. It allows you to visually compose data transformation workflows using nodes that represent different data handling steps, which later are converted automatically into code to run.

Visualization 73

Visualization Marketing Big Data IT 73

AWS Big Data

MARCH 26, 2024

This view is used to identify patterns and trends in customer behavior, which can inform data-driven decisions to improve business outcomes. In this post, we discuss how you can use purpose-built AWS services to create an end-to-end data strategy for C360 to unify and govern customer data that address these challenges.

Data Strategy 67

Data Strategy Strategy Data Warehouse Prescriptive Analytics 67

AWS Big Data

JUNE 6, 2023

AWS Glue Data Quality allows you to measure and monitor the quality of data in your data repositories. It’s important for business users to be able to see quality scores and metrics to make confident business decisions and debug data quality issues. An AWS Glue crawler crawls the results.

Data Quality 70

Data Quality Metrics Visualization Dashboards 70

Jen Stirrup

JANUARY 1, 2023

In this post, I’ll share some New Year Resolutions that I think Business Intelligence, Data Engineering, Artificial Intelligence, and Data Science teams should start doing from today. I get to see both sides; the business and the technology. Adopt robust Data Engineering Practices. ’ ( Reference ).

Business Intelligence 62

Business Intelligence Data Science Consulting Software 62

AWS Big Data

APRIL 20, 2023

In a modern data architecture, unified analytics enable you to access the data you need, whether it’s stored in a data lake or a data warehouse. One of the most common use cases for data preparation on Amazon Redshift is to ingest and transform data from different data stores into an Amazon Redshift data warehouse.

Visualization 77

Visualization Data Warehouse Big Data Data Lake 77

AWS Big Data

OCTOBER 3, 2023

A data lake is a centralized repository that you can use to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data and then run different types of analytics for better business insights. Analytics use cases on data lakes are always evolving.

Data Lake 100

Data Lake Metadata Snapshot Recreation/Entertainment 100

AWS Big Data

APRIL 28, 2023

Amazon Redshift is a fast, scalable, secure, and fully managed cloud data warehouse that makes it simple and cost-effective to analyze all your data using standard SQL and your existing ETL (extract, transform, and load), business intelligence (BI), and reporting tools. The following diagram illustrates this architecture.

Interactive 89

Interactive Metadata Data Warehouse Data-driven 89

Domino Data Lab

AUGUST 20, 2019

Many thanks to AWP Pearson for the permission to excerpt “Manual Feature Engineering: Manipulating Data for Fun and Profit” from the book, Machine Learning with Python for Everyone by Mark E. Feature engineering is useful for data scientists when assessing tradeoff decisions regarding the impact of their ML models.

Testing 68

Testing Modeling Interactive Measurement 68

O'Reilly on Data

FEBRUARY 11, 2020

We suspected that data quality was a topic brimming with interest. The responses show a surfeit of concerns around data quality and some uncertainty about how best to address those concerns. Key survey results: The C-suite is engaged with data quality. Data quality might get worse before it gets better.

Data Quality 304

Data Quality Metadata Data Governance Publishing 304

Cloudera

AUGUST 17, 2021

When we announced the GA of Cloudera Data Engineering back in September of last year, a key vision we had was to simplify the automation of data transformation pipelines at scale. And by being purely python based, Apache Airflow pipelines are accessible to a wide range of users, with a strong open source community.

Management 110

Management Cost-Benefit Data Transformation Optimization 110

O'Reilly on Data

SEPTEMBER 11, 2019

Machine learning, artificial intelligence, data engineering, and architecture are driving the data space. The Strata Data Conferences helped chronicle the birth of big data, as well as the emergence of data science, streaming, and machine learning (ML) as disruptive phenomena. The term “AI,” meanwhile, is No.

IoT 20

IoT Big Data Data Warehouse Uncertainty 20

O'Reilly on Data

DECEMBER 19, 2018

We need to do more than automate model building with autoML; we need to automate tasks at every stage of the data pipeline. In a previous post , we talked about applications of machine learning (ML) to software development, which included a tour through sample tools in data science and for managing data infrastructure.

Machine Learning 261

Machine Learning Software Modeling Testing 261