Since initial support was added in Apache Spark 2.3, running Spark on Kubernetes has been growing in popularity. Reasons include the improved isolation and resource sharing of concurrent Spark applications on Kubernetes, as well as the benefit to use an homogeneous and cloud native infrastructure for the entire tech stack of a company. But running Spark on Kubernetes in a stable, performant, cost-efficient and secure manner also presents specific challenges. In this talk, JY and Julien will go over lessons learned while building Data Mechanics, a serverless Spark platform powered by Kubernetes.

Topics include:

- Core concepts and setup of Spark on Kubernetes
- Configuration tips for performance and efficient resource sharing
- Spark-app level dynamic allocation and cluster level autoscaling
- Specificities of Kubernetes for data I/O performance
- Monitoring and security best practices
- Limitations and planned future works

About:
Databricks provides a unified data analytics platform, powered by Apache Spark™…

After several months spent as an "experimental" feature in Apache Spark, Kubernetes was officially promoted to a Generally Available scheduler in the 3.1 release! In this blog post, we'll discover the last changes made before this promotion.

Participate in a live poll, quiz or Q&A. No login required.