KDnuggets

FEBRUARY 22, 2024

Top list of open-source tools for building and managing workflows.

Management 119

Management 119

AWS Big Data

APRIL 25, 2024

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed orchestration service for Apache Airflow that you can use to set up and operate data pipelines in the cloud at scale. By using multiple AWS accounts, organizations can effectively scale their workloads and manage their complexity as they grow.

Metadata 101

Metadata Data Processing Management Testing 101

AWS Big Data

NOVEMBER 6, 2023

Amazon Managed Workflow for Apache Airflow (Amazon MWAA) is a managed service that allows you to use a familiar Apache Airflow environment with improved scalability, availability, and security to enhance and scale your business workflows without the operational burden of managing the underlying infrastructure.

Metrics 100

Metrics Metadata Snapshot Management 100

Cloudera

SEPTEMBER 21, 2021

Many customers looking at modernizing their pipeline orchestration have turned to Apache Airflow, a flexible and scalable workflow manager for data engineers. Apache Airflow providers are a set of packages allowing services to define operators in their Directed Acyclic Graphs (DAGs) to access external systems.

Data Processing 99

Data Processing Testing Management Machine Learning 99

AWS Big Data

APRIL 17, 2023

We have built data pipelines to process, aggregate, and clean our data for our forecasting service. With the growing interest in our services, we wanted to scale our batch-based data pipeline to process more historical data on a daily basis and yet remain performant, cost-efficient, and predictable.

Forecasting 76

Forecasting Cost-Benefit Optimization Testing 76

AWS Big Data

OCTOBER 31, 2023

Customers rely on data from different sources such as mobile applications, clickstream events from websites, historical data, and more to deduce meaningful patterns to optimize their products, services, and processes. Apache Airflow and Snowflake have emerged as powerful technologies for data management and analysis.

Data Processing 87

Data Processing Management Publishing Testing 87

Cloudera

SEPTEMBER 17, 2020

For enterprise organizations, managing and operationalizing increasingly complex data across the business has presented a significant challenge for staying competitive in analytic and data science driven markets. CDP data lifecycle integration and SDX security and governance. Enterprise Data Engineering From the Ground Up.

Visualization 104

Visualization Metrics Statistics Optimization 104