AWS Big Data

NOVEMBER 17, 2023

Managing the SQL files, integrating cross-team work, incorporating all software engineering principles, and importing external utilities can be a time-consuming task that requires complex design and lots of preparation. In this post, we look into an optimal and cost-effective way of incorporating dbt within Amazon Redshift.

Snapshot 87

Snapshot Data Processing Testing Data Warehouse 87

DataKitchen

JULY 27, 2023

Azure Databricks Workflows : An Apache Spark-based analytics platform optimized for the Microsoft Azure cloud services platform. SQL Server Integration Services (SSIS): You know it; your father used it. You can use it for big data analytics and machine learning workloads. Workflows is a DAG runner embedded in Databricks.

Machine Learning 130

Machine Learning Cost-Benefit Data Transformation Testing 130

DataKitchen

APRIL 13, 2021

Airflow — An open-source platform to programmatically author, schedule, and monitor data pipelines. Apache Oozie — An open-source workflow scheduler system to manage Apache Hadoop jobs. DBT (Data Build Tool) — A command-line tool that enables data analysts and engineers to transform data in their warehouse more effectively.

Testing 307

Testing Machine Learning Consulting Data Quality 307