Interview prep library
Data engineering topics, one place
Your original 12 questions merged with the extended list into 79 topics across seven areas. Tick topics off as you go; progress stays in this browser. Start with the Spark Core path below.
Start here
Spark Core path
A short background page and five modules, in order. Each module starts with a quick overview; deeper pages hang off it. Plan about 45–60 minutes per module.
HDFS, YARN, MapReduce, and what Spark solved. Optional context.
Driver, executors, jobs/stages/tasks, the memory model. Deep dive on dynamic allocation and speculation; production memory failures.
Where each runs, why DataFrames win, Spark vs pandas, toPandas and Arrow.
spark-submit to completion, plan building, narrow vs wide, jobs → stages → tasks.
Optimizer phases, pushdown and pruning, reading explain(), join strategies, AQE, UDF cost.
Debug flow, six failure scenarios, shuffle reduction, partition and executor sizing.
Suggested order after Spark Core: Structured Streaming → Delta Lake & Databricks → Iceberg → Modelling & SQL → Pipelines → Governance. Ask in chat and the next path gets built into this hub.
Tracker