Spark Core · Module 2 of 5 · Overview

RDD, DataFrame, Dataset and pandas

Five ways to hold tabular data when you work with PySpark: where each one runs, why DataFrames are faster than RDDs, and when pandas is the right tool. A 10-minute refresher.

The big picture Side by side Why DataFrames win When RDDs still fit Spark vs pandas Interview answers Quick check Go deeper

The big picture

The RDD is Spark's original, low-level API: a distributed collection of objects that you transform with your own functions. DataFrames (and Datasets in Scala and Java) add a schema of named, typed columns, which lets Spark understand and optimise your query. Today they're the default; RDDs still run underneath as the execution engine.

pandas is a different world: a single-machine, in-memory, eager library. It's excellent for data that fits on one machine, and the two worlds meet through toPandas(), pandas UDFs and the pandas API on Spark.

Everything on the left is distributed and planned by Catalyst before it runs, except RDD code you write directly. pandas on the right lives in one process. Moving data across the gap is where most memory problems start.

Side by side

	RDD	DataFrame	Dataset[T]	pandas DataFrame	pandas API on Spark
Runs on	Cluster	Cluster	Cluster	One machine	Cluster
Evaluation	Lazy	Lazy	Lazy	Eager	Lazy (most operations)
Structure	Opaque objects, no schema	Named columns; types checked when the query is analysed	Typed objects; types checked at compile time	Columns plus an index	Columns plus an index
Optimised by Catalyst	No	Yes	Column expressions yes; typed lambdas only partly	No optimiser	Yes (translated to DataFrame plans)
Languages	Scala, Java, Python	Python, Scala, Java, R, SQL	Scala, Java	Python	Python
Mutable?	No	No	No	Yes	Pandas-like syntax; immutable underneath
Data size	Any	Any	Any	Must fit in one machine's memory	Any
Best for	Low-level control, unstructured data, legacy code	Almost everything	Typed Scala/Java pipelines	Small data, local analysis, ML and plotting libraries	pandas users moving to big data

In Scala and Java, a DataFrame is a Dataset[Row]. Python and R have no Dataset API because they have no compile-time types to check.

Why DataFrames beat RDDs

Spark can see what you mean. With filter(col("country") == "AE") Catalyst knows the column and the condition, so it can push the filter into the Parquet reader and read only the columns you use. An RDD lambda is a black box: Spark must read every row and every column and hand them to your function.
Compact memory format. Tungsten stores DataFrame rows as packed binary data, not one JVM object per row, which means less memory, less garbage collection and generated code that runs close to the CPU.
In PySpark the gap is largest. DataFrame operations run entirely inside the JVM. RDD lambdas written in Python force every row to be serialised (pickled), sent to a Python worker process, processed, and sent back. See it animated →

Spark SQL and the DataFrame API compile to the same plan, so they perform the same. The choice between them is style, not speed.

When RDDs still fit

Truly unstructured data where no schema makes sense, or record parsing that needs full control.
Fine-grained control over partitioning or physical placement that the DataFrame API doesn't expose.
Maintaining legacy code. For new code, mapInPandas or applyInPandas usually covers "I need my own function" cases with better performance.

Modern platforms are dropping RDD access

Spark Connect (Spark 3.4+) and Databricks serverless and shared-access compute don't support the RDD API or sparkContext. Code that relies on df.rdd won't run there. One more reason to stay on DataFrames.

Spark DataFrame vs pandas DataFrame

	Spark DataFrame	pandas DataFrame
Scale	Distributed across executors; any size	One process; limited by that machine's RAM
Execution	Lazy: builds a plan, runs on an action	Eager: every line runs immediately
Changes	Immutable: each transformation returns a new DataFrame	Mutable: `df["x"] = …` changes it in place
Row order and index	No index; row order isn't guaranteed without `orderBy`	Labelled index; order is preserved
Where pandas code runs in a Spark job	After `toPandas()`: on the driver, with all the data. Inside a pandas UDF: on executors, one batch or group at a time.

Say it in an interview

RDD vs DataFrame vs Dataset

"An RDD is a distributed collection of objects transformed with arbitrary functions, so Spark can't optimise it. A DataFrame adds a schema, so Catalyst can push down filters, prune columns, choose joins and Tungsten can store rows compactly. A Dataset is the typed version for Scala and Java, with compile-time checks; a DataFrame is Dataset[Row]. I use DataFrames by default and RDDs only for low-level cases."

Why DataFrames are faster in PySpark

"DataFrame operations stay in the JVM and are optimised as a whole plan. Python RDD lambdas send every row through pickling to Python worker processes and back, and Spark can't see inside them to push down filters or prune columns."

Spark vs pandas

"Spark DataFrames are distributed, lazy and immutable; pandas DataFrames are single-machine, eager and mutable. toPandas collects everything to the driver, so I use it only on small results, with Arrow enabled. For pandas logic at scale I use pandas UDFs or the pandas API on Spark."

Quick check

Why is there no Dataset API in PySpark?

Datasets exist to give compile-time type safety through JVM classes. Python is dynamically typed, so there's nothing to check at compile time; a PySpark DataFrame already covers the same ground.

A colleague converts a DataFrame to an RDD to apply a Python lambda, then back with toDF(). What's lost?

Optimisation across that boundary (no pushdown or pruning through the lambda), Tungsten's compact format, and speed, since every row is pickled to Python and back. Built-in functions, or a pandas UDF / mapInPandas, avoid most of that. It also won't run on Spark Connect.

Is a Spark SQL query faster than the same DataFrame code?

No. Both produce the same logical plan and go through Catalyst identically.

Where does pandas code run in a Spark application?

After toPandas(), on the driver, with the full result in memory. Inside a pandas UDF, mapInPandas or applyInPandas, on executors, one Arrow batch or group at a time.

Is the pandas API on Spark just pandas running on the driver?

No. It translates pandas-style calls into Spark DataFrame plans that run distributed. Only to_pandas() brings data to the driver.

Go deeper

Deep diveArrow and vectorised execution

Row vs columnar memory, what Apache Arrow does at the JVM ↔ Python boundary, why pandas UDFs are faster, and the three meanings of "vectorised". Two interactive diagrams.

ExamplesOne query, six ways

The same aggregation in RDD, DataFrame, SQL, Scala Dataset, pandas and the pandas API on Spark, with what Spark can and can't optimise in each, plus every conversion between them.

Production issuesCrossing between Spark and pandas

toPandas OOM, silent Arrow fallback, slow Python UDFs, losing optimisation through RDDs, pandas-on-Spark surprises, and RDD code failing on Spark Connect.