Cloudera

MAY 5, 2021

Apache YuniKorn (Incubating) has just released 0.10.0 ( release announcement ). By leveraging the Gang Scheduling feature, Spark jobs scheduling on Kubernetes becomes more efficient. What is Apache YuniKorn (Incubating)? Gang Scheduling in YuniKorn. What is Gang Scheduling?

Metadata 135

Metadata Machine Learning Big Data IT 135

Cloudera

SEPTEMBER 2, 2021

Normally on-premises, one of the key challenges was how to allocate resources within a finite set of resources (i.e., When building CDE, we integrated with Apache YuniKorn which offers rich scheduling capabilities on Kubernetes. . fixed sized clusters). The traditional scaling pattern can be observed on the graph below.

Optimization 81

Optimization Testing Cost-Benefit Measurement 81

Cloudera

OCTOBER 14, 2020

Why choose K8s for Apache Spark. Apache Spark unifies batch processing, real-time processing, stream analytics, machine learning, and interactive query in one-platform. Kubernetes as a de-facto standard for service deployment offers finer control on all of the above aspects compared to other resource orchestrators.

Machine Learning 112

Machine Learning Management Big Data Optimization 112