DE Learning

Spark Core · Module 1 of 5 · Overview

Who does what in a Spark application

A 10-minute refresher: the parts of a Spark application, the units of work they run, and how executor memory is divided. Use the deep-dive and production pages when you need more.

ComponentsJobs, stages, tasksMemoryKey factsInterview answersQuick checkGo deeper

The three moving parts

A Spark application is one driver plus the executors it gets from a cluster manager. The driver plans the work and schedules tasks. Executors are JVM processes on worker nodes that run those tasks, one per core, and hold shuffle files and cached data. The cluster manager only hands out machines; it doesn't schedule tasks.

Spark architecture: driver, cluster manager and two worker nodes with executors Driverone JVM, plans the work Your application codemain(), DataFrame API, SQL SparkSessionentry point, catalog, config DAG schedulersplits a job into stages at shuffles Task schedulersends one task per partition to a free core Cluster manager YARN · Kubernetes Standalone · Databricks Worker node 1 Executor (JVM)4 cores = 4 slots task task task idle cached blocks · shuffle files on local disk Worker node 2 Executor (JVM)4 cores = 4 slots task task idle idle cached blocks · shuffle files on local disk 1 2 2 3–5 3–5
  1. The driver asks the cluster manager for executors (fixed count, or dynamic allocation).
  2. The cluster manager starts executor JVMs on worker nodes with the requested cores and memory.
  3. Executors register back with the driver. From now on the driver talks to them directly.
  4. For each stage, the task scheduler sends tasks (your serialised code plus one partition to work on) to free cores, preferring executors that already hold that data.
  5. Executors report status and send small results back. Shuffle files and cached data stay on the executors.

Jobs, stages and tasks

An action creates a job, shuffles split the job into stages, and each stage runs one task per partition on executor cores. The driver does all the planning; executors only ever see tasks.

Application   one SparkSession: driver + executors
└─ Job        one per action (count, write, collect)    ← created by the driver
   └─ Stage   a new one at every shuffle                ← cut by the DAG scheduler
      └─ Task one per partition, one core each          ← run by executors

Module 3 walks a real pipeline through this: Jobs, stages and tasks →

Executor memory at a glance

Think of an executor as a shared office floor. The cluster manager rents you the whole floor (the container). Spark carves it into fixed areas, and one area, the unified pool, is shared by every task running on that executor's cores. Example: spark.executor.memory=16g, 4 cores, defaults.

Container (what YARN / Kubernetes allocates)             17.6 GB
├─ Overhead, off-heap: JVM internals, network, Python      1.6 GB   max(384 MB, 10% of heap)
└─ JVM heap = spark.executor.memory                        16 GB
   ├─ Reserved for Spark internals                         0.3 GB   fixed
   ├─ User memory: your objects, UDF data structures       6.3 GB   40% of the rest
   └─ Unified pool                                         9.4 GB   60% of the rest
      ├─ Storage: cache, broadcast variables              ~4.7 GB   ┐ soft boundary,
      └─ Execution: shuffle, join, sort, aggregate        ~4.7 GB   ┘ can move
  • Execution can evict cache (down to the protected half), but cache can't evict execution. Losing cache only costs a recompute; losing a running join's memory would break the task.
  • Memory per task ≈ pool ÷ cores. Here about 2.35 GB per task with no cache, 1.2 GB if cache holds its half. Going past that means spill (slow) or OutOfMemoryError.
  • "Java heap space" means the heap ran out. "Container killed … exceeding memory limits" means overhead ran out (often Python). Different causes, different fixes.

Key facts

FactRule or default
SlotOne executor core runs one task at a time. Total slots = executors × cores.
WavesTasks ÷ slots. Stage time ≈ waves × task time, and never less than the slowest task.
Deploy modeClient: driver on the submitting machine (interactive). Cluster: driver inside the cluster (production).
Reserved memory300 MB, fixed
spark.memory.fraction0.6 of (heap − 300 MB) forms the unified pool
spark.memory.storageFraction0.5 of the pool is protected for cache
spark.executor.memoryOverheadmax(384 MB, 10% of heap), outside the heap
Memory per task≈ unified pool ÷ cores per executor
Dynamic allocationAdds executors when tasks wait over 1 s; removes them after 60 s idle. Needs a shuffle service or shuffle tracking.
SpeculationOff by default. Re-runs a task slower than 1.5 × median once 75% of the stage is done. Doesn't fix skew.

Say it in an interview

Architecture
"The driver builds the plan and schedules tasks; executors are JVMs on worker nodes that run those tasks in parallel, one per core, and hold shuffle files and cached data. The cluster manager only allocates resources."
Jobs, stages, tasks
"An action creates a job. The DAG scheduler starts a new stage at every shuffle, so stages are runs of narrow transformations. Each stage runs one task per partition, so partitions decide parallelism."
Memory
"Executor memory is the heap plus off-heap overhead. After 300 MB reserved, 60% of the heap is a unified pool shared by execution and storage, where execution can evict cache but not the reverse. Memory per task is roughly that pool divided by cores, and that decides whether a task spills or fails."

Quick check

Does the cluster manager schedule tasks?
No. It only allocates containers or pods for executors. The driver's DAG scheduler and task scheduler decide what runs where.
10 executors × 4 cores, a stage of 1,000 tasks taking 20 s each. How long?
40 slots, so 25 waves × 20 s ≈ 500 s, roughly 8–9 minutes. More slots is the main lever; one skewed task sets a floor no cluster size can beat. See the wave timeline →
"Container killed by YARN for exceeding memory limits." Heap or overhead?
Overhead. The process as a whole went over the container limit, usually because of Python workers or off-heap buffers. Raise spark.executor.memoryOverhead or reduce Python memory use. Full scenario →
A 16 GB, 4-core executor spills heavily. Two fixes that don't add memory?
Use more shuffle partitions so each task handles less data, or drop to 2 cores per executor so each task gets about twice the memory. Check for skew first.
Does speculative execution fix data skew?
No. A skewed task is slow because its partition is bigger. The copy gets the same partition and is just as slow. Speculation only helps when the machine is the problem. Why →

Go deeper

Deep diveSlots, memory internals, dynamic allocation & speculation

Wave timeline, the memory layout drawn to scale with a calculator, how executors are added and removed, how stragglers get a backup copy, and deploy modes.

 Production issuesExecutor memory: failures and fixes

Eight common memory failures, each with the real error, how to confirm it, the fix, prevention and an interview answer. Plus a mitigation cheat sheet.

Mark this module's topics