Cloudera

MAY 9, 2022

Tuning Hive on Tez queries can never be done in a one-size-fits-all approach. The performance on queries depends on the size of the data, file types, query design, and query patterns. During performance testing, evaluate and validate configuration parameters and any SQL modifications. Understanding parallelization in Tez.

Optimization 114

Optimization Testing Cost-Benefit Measurement 114

Cloudera

SEPTEMBER 28, 2020

In addition the customer wanted to use the new Hive capabilities shipped with CDP Private Cloud Base 7.1.2. Hive-on-Tez for better ETL performance. ACID transactions, ANSI 2016 SQL SupportMajor Performance improvements. Navigator to atlas migration, Improved performance and scalability. Background: .

Testing 130

Testing Metadata Risk Data Science 130

AWS Big Data

MARCH 12, 2024

Our commitment to efficiency is unwavering, and we’ve undertaken an exciting initiative to optimize our batch processing jobs. Using best practices learned from the AWS FinHack program, we fine-tuned resource-intensive jobs, converted Pig and Hive jobs to Spark, and reduced our batch workload spend by 22.75% in 2022.

Cost-Benefit 91

Cost-Benefit Optimization Big Data Metrics 91

AWS Big Data

FEBRUARY 6, 2023

Additionally, a TCO calculator generates the TCO estimation of an optimized EMR cluster for facilitating the migration. In this post, we dive deep into the tool, walking through all steps from log ingestion, transformation, visualization, and architecture design to calculate TCO. Now let’s look at how the tool works.

Dashboards 74

Dashboards Optimization Data Lake Cost-Benefit 74

AWS Big Data

FEBRUARY 6, 2023

When migrating Hadoop workloads to Amazon EMR , it’s often difficult to identify the optimal cluster configuration without analyzing existing workloads by hand. The optimized future EMR cluster yields the same results and values with much lower TCO compared to the source Hadoop cluster.

Cost-Benefit 52

Cost-Benefit Data Lake Dashboards Big Data 52

Cloudera

AUGUST 17, 2021

That’s why we are excited to expand our Apache Airflow-based pipeline orchestration for Cloudera Data Platform (CDP) with the flexibility to define scalable transformations with a combination of Spark and Hive. Figure 1: Pipeline composed of Spark and Hive jobs deployed to run within CDE’s managed Apache Airflow service.

Management 110

Management Cost-Benefit Data Transformation Optimization 110

AWS Big Data

MARCH 3, 2023

A job is run in a single Availability Zone to avoid performance implications of network traffic across Availability Zones. EMR Serverless automatically provisions and scales the compute and memory resources required by your applications, and you only pay for the resources that the applications use.

Data Lake 81

Data Lake Dashboards Metrics Metadata 81