The interview process started with a recruiter screening discussing my background, Spark/PySpark experience, SQL skills, and large-scale ETL pipeline development. The next round was a technical interview focused on Python, SQL, data modeling, and distributed data processing concepts. I was asked about Spark optimizations, partitioning, joins, and handling large datasets in production environments. There was also discussion around real-world data pipeline troubleshooting and system design. T

Interviewed with Lead engineer on spark although recruiter said it was on python. Not a big deal. Asked to for a group by aggregation. Did a groupby and sum and joined results. Asked if I could do it without a join. Did it with a sum over a partition. Asked if there was another way. Added a withColumn statement and put groupby statement as 2nd argument. Interviewer told me it was starting to look good although that's not possible in pyspark. recruiter let me know I failed tech screen.