Spark Core · Background · Overview
Hadoop and MapReduce, and what Spark changed
Spark was built to fix MapReduce's limits, and many of its ideas still run inside Spark today. A 10-minute refresher for the "Hadoop basics" and "what did Spark solve?" questions.
Hadoop's three partsHow a MapReduce job runsKey termsWhat Spark solvedWhat lives on in SparkInterview answersQuick check
Hadoop's three core components
Hadoop splits the problem in three: where data is stored, who hands out machines, and how computation is expressed. Spark replaced only the third. It still commonly reads from HDFS-style storage and still runs on YARN.
| Component | Role | Main pieces |
|---|---|---|
| HDFS | Distributed storage: files split into large blocks, spread across machines | NameNode holds metadata (which blocks make up a file, where they are). DataNodes store blocks, 128 MB by default, each replicated 3 times. |
| YARN | Resource management: decides which application gets which CPU and memory | ResourceManager (cluster-wide), NodeManager (per machine), and one ApplicationMaster per job. Work runs in containers. |
| MapReduce | Processing model: express a job as a map phase and a reduce phase | Map tasks process input splits in parallel; reduce tasks aggregate their output by key. |
How a MapReduce job runs: word count
The classic example: count how often each word appears across many files.
- Split. The InputFormat divides input files into input splits, usually one per HDFS block. One map task per split, scheduled near the data where possible (data locality).
- Read records. A RecordReader turns the raw bytes of a split into key-value records. For text, key = byte offset, value = one line.
- Map. Your
map()runs once per record and emits intermediate key-value pairs: here(word, 1). - Combine, partition, sort. An optional Combiner pre-aggregates on the map side. A Partitioner (hash of the key by default) picks the reducer for each key. Output is sorted and spilled to the mapper's local disk.
- Shuffle. Each reducer fetches its partitions from every mapper over the network and merge-sorts them by key.
- Reduce. Your
reduce()runs once per key with all its values, and writes final output to HDFS, replicated 3 times.
Anything more complex than one map and one reduce, such as a join followed by an aggregation, becomes a chain of separate jobs. Each job writes its full result to HDFS and the next one reads it back.
Key terms
| Term | What it does | Worth knowing |
|---|---|---|
| InputSplit | A logical chunk of input for one map task | Usually matches an HDFS block; it's a range, not a copy of the data |
| RecordReader | Turns a split's bytes into key-value records for the mapper | Handles records that cross split boundaries (a line split across two blocks is read whole by one mapper) |
| Combiner | A "mini-reducer" on each mapper's output before the shuffle | Cuts network traffic. Optional, and may run zero, one or several times, so the function must be associative and commutative (sum, max: yes; average: not directly) |
| Partitioner | Decides which reducer receives each key | Default is hash(key) mod reducers; a hot key overloads one reducer, the same skew problem as in Spark |
| Shuffle & sort | Moves map output to reducers, grouped and sorted by key | The most expensive phase: disk, network and sorting |
What Spark solved
| MapReduce limitation | Spark's answer |
|---|---|
| Every job writes its full result to HDFS (replicated) and the next job reads it back; iterative work (ML, graphs) pays this each iteration | In-memory processing: intermediate data stays in executor memory, and reused data can be cached |
| Rigid map → reduce shape; complex logic needs a chain of jobs | General DAG: one job can have many stages, and narrow steps are pipelined together |
| Each task starts a new JVM, giving high startup latency | Long-lived executors run tasks as threads, so tasks start in milliseconds |
| Low-level, verbose Java API | High-level APIs: DataFrames, SQL, Python, Scala, R |
| No query optimiser; performance depends on hand-written code | Catalyst and Tungsten optimise and compile the plan |
| Batch only; separate systems for SQL, streaming, ML | One engine for batch, SQL, streaming and ML |
Common trap: "Spark is in-memory, so it never touches disk"
Not true. Spark shuffles still write map output to local disk, just like MapReduce, and tasks spill to disk when memory runs out. The real gain is avoiding HDFS round-trips between steps, plus caching, pipelining and fast task start-up.What lives on in Spark
| MapReduce idea | In Spark | Where we cover it |
|---|---|---|
| Map phase | Narrow transformations (map, filter, select), pipelined in one stage | Execution |
| Shuffle & sort | Wide transformations; an Exchange in the plan; a stage boundary | Execution |
| Combiner | Map-side combine: reduceByKey, and partial_sum before the shuffle in DataFrame plans | reduceByKey vs groupByKey |
| Partitioner | hashpartitioning(key, n); spark.sql.shuffle.partitions | Partition sizing |
| Data locality | Task scheduler prefers executors that hold the data | Architecture |
| YARN containers | Spark executors run in YARN containers (or Kubernetes pods) | Executor memory |
Say it in an interview
Hadoop components
"HDFS stores data as replicated 128 MB blocks, with a NameNode for metadata and DataNodes for blocks. YARN allocates resources through a ResourceManager, per-node NodeManagers and a per-job ApplicationMaster. MapReduce is the processing model: map tasks over input splits, a shuffle and sort by key, then reduce tasks."What Spark solved
"MapReduce writes every job's output to HDFS and forces a rigid map-then-reduce shape, so multi-step and iterative work is slow. Spark keeps intermediate data in memory, runs a whole DAG of stages in one job, reuses long-lived executors, and adds high-level APIs with an optimiser. It still writes shuffle files to local disk, though, so it isn't purely in-memory."Quick check
What are the three core components of Hadoop?
HDFS (storage), YARN (resource management) and MapReduce (processing).
What does a RecordReader do?
It converts the bytes of an input split into key-value records for the mapper, for example byte offset → line for text files, and handles records that cross split boundaries.
Can you use a Combiner to compute an average?
Not directly: an average of averages is wrong when groups differ in size. Emit (sum, count) pairs instead; those combine correctly, and the reducer divides at the end. The same trick applies to Spark's
aggregateByKey.Name three MapReduce limitations Spark addressed.
HDFS writes between every job, the rigid two-phase model that forces job chains, and JVM start-up per task. Also: no optimiser, a verbose API, and batch-only processing.
Does Spark ever write intermediate data to disk?
Yes. Shuffle output always goes to the executors' local disks, and tasks spill to disk under memory pressure. What Spark avoids is writing every intermediate result to replicated HDFS.