A developer on the team wrote an ETL that runs once a day as a Spark job. Every day it reads a CSV file that shows the total value of each customer's transactions of that day and writes them as a parquet file partitioned by date and customer id. Below you can see an example of the CSV file. Note that each customer has one entry representing the total transaction value it did on that day. However, sometimes the CSV file contains a correction for a sum reported in the past. For example - this file represents the transactions on 1/10. You can see that customer 1002 has 2 entries. One for 1/10 and one for 30/9. This means that the total sum of transactions the customer did on 1/10 is 70, but the total sum of transactions it did on 30/9 was 40 and this sum should replace the value already reported on 30/9. current date file: 2020-10-01 date,customer,price 2020-10-01,1000,40 2020-10-01,1001,10 2020-09-30,1002,40 2020-10-01,1002,70 2020-10-01,1003,10 2020-09-29,1004,10 2020-10-01,1004,10 This function represents the ETL. It runs once a day with a string representing the current day. It reads the CSV file, does some transformations, and writes it. Please help us find the bug in the code above, and return the right results

A developer on the team was running the following line in a function for logging purposes, and the job crashed with an "out of memory" exception. The developer says that the cluster has many workers with a lot of memory and disk and still the job crashes. Can you help explain how come to this line makes the job crash with OOM even though the cluster is huge? def someFunc(): for row in df.collect(): print(f'Customerr{row["customer"]} => Paid {row["price"]}')

Our developer had to join the results with a dimensional table of categories. The join works, but its a bit slow, see if you can understand why and whether it can run faster