Spark Core · Module 3 of 5 · Overview

From your code to running tasks

How Spark turns Python into work on the cluster: transformations build a plan, actions run it, shuffles cut it into stages, and partitions become tasks. A 10-minute refresher.

The flow Lazy evaluation Narrow vs wide Jobs, stages, tasks Key facts Interview answers Quick check Go deeper

The flow in five steps

Start. spark-submit (or a notebook) starts the driver, which gets executors from the cluster manager.
Describe. Your code runs on the driver. Each transformation only adds a step to a logical plan; no data moves.
Trigger. An action (write, count, collect, show) creates a job.
Plan. Catalyst optimises the plan; the DAG scheduler cuts it into stages at every shuffle.
Run. Each stage runs one task per partition on executor cores; stages run in dependency order until the job ends.

All ten steps, including retries and data locality: Deep dive →

Lazy evaluation: per action, not per script

Transformations don't run anything; they extend a DataFrame's plan (its lineage). Your Python still runs line by line. When the interpreter reaches an action, that line blocks: Spark optimises that DataFrame's lineage so far, runs it as a job, and returns. Spark never reads ahead in your script.

orders = spark.read.parquet("s3://lake/orders")    # metadata only
ae     = orders.filter("country = 'AE'")           # plan grows, instant
n      = ae.count()                                # Job 1 runs now; Python waits
if n > 0:                                          # plain Python
    ae.groupBy("segment").sum("amount") \
      .write.saveAsTable("gold.sales")             # Job 2: re-reads orders from scratch

Because it's lazy, Spark can optimise a whole chain at once (push filters into the scan, fuse narrow steps). Because it's per action, every action recomputes its lineage unless you cache. Timeline and the exceptions →

Narrow vs wide transformations

A narrow transformation builds each output partition from one input partition, so no data moves between machines. A wide one needs rows from many partitions, so Spark shuffles data across the network, and that's where a new stage starts.

Narrow (no shuffle)	Wide (shuffle)	It depends
`select`, `filter`, `withColumn`, `map`, `flatMap`, `union`, `coalesce`, `explode`	`groupBy`/`agg`, `distinct`, `dropDuplicates`, `orderBy`, `repartition`, sort-merge `join`, window `partitionBy`	`join` is narrow for the big side when the other side is broadcast, or both sides are already partitioned the same way

Jobs, stages and tasks

One action → one job. The DAG scheduler walks the plan back from the result and starts a new stage at every shuffle (shown as Exchange in plans). Each stage runs one task per partition. Here's one job that filters orders, joins customers, sums by segment and writes the result:

One action → one job → four stages. If customers is under the broadcast threshold (10 MB by default), Stage 1 becomes a broadcast, the orders side is not shuffled, and Stages 0 and 2 merge into one.

Key facts

Fact	Rule or default
Job	One per action (plus small extra jobs for schema inference, file listing or broadcasts)
Stage boundary	Every shuffle (`Exchange`); narrow steps either side are pipelined into one stage
Tasks per stage	= partitions in that stage
Scan partitions	About one per 128 MB of input (`spark.sql.files.maxPartitionBytes`)
Post-shuffle partitions	`spark.sql.shuffle.partitions` = 200, unless AQE coalesces them
Task retries	`spark.task.maxFailures` = 4, then the job fails
Lost shuffle output	The parent stage's missing tasks are recomputed from lineage
Recomputation	Every action re-runs its lineage from the source unless cached

Say it in an interview

Lazy evaluation

"Python executes line by line on the driver. Transformations only extend a DataFrame's logical plan; when the interpreter reaches an action, Spark optimises that lineage, runs it as one or more jobs and blocks until it returns. It never looks ahead, so each action recomputes its lineage unless it's cached."

Jobs, stages, tasks

"An action creates a job. The DAG scheduler starts a new stage at every shuffle dependency, so stages are runs of narrow transformations pipelined together. Each stage runs one task per partition, so partitions decide parallelism."

Narrow vs wide

"Narrow transformations like filter and select work within a partition and are pipelined in one stage. Wide ones like groupBy, distinct and most joins need rows from many partitions, so they shuffle data over the network and create a stage boundary. A broadcast join avoids shuffling the large side."

Quick check

How many jobs: df = spark.read.parquet(p); df.count(); df.filter(...).write.parquet(q)?

At least two, one per action. The read may add a small listing job. Both actions read the files, unless df is cached.

Where exactly is a stage boundary created?

At every wide (shuffle) dependency, an Exchange in the physical plan. Narrow operations either side are pipelined into the same stage.

Is coalesce narrow or wide?

Narrow: it merges existing partitions without a shuffle. repartition is wide.

Does Spark run your whole script lazily as one plan?

No. Laziness is per DataFrame. Python runs normally, and each action runs immediately, blocking until its job finishes.

What happens when a task fails?

It's retried, up to 4 attempts by default. If shuffle files from an earlier stage were lost with an executor, the missing parent tasks are recomputed from lineage first.

Go deeper

Deep diveHow plans become stages

All ten submit steps, a Python-vs-cluster timeline, calls that run jobs eagerly, lineage and immutability, how stages are cut, skipped stages, reduceByKey vs groupByKey.

ExamplesStep through it, count the jobs

An interactive lazy-evaluation stepper, plus six code snippets to work out jobs and stages yourself.

Production issuesWhen execution bites

Recomputed lineage, leftover debug actions, huge plans from loops, fetch failures, aborted jobs, and errors that surface far from their cause.