Скачать или смотреть How to set your Spark Configs right with OpenCostsInBytes - Spark Partitioning (Part 10)

OpenCostsInBytes is one of the configs Spark tries to deal with its small file problem. What it does: It's patting incoming files with a defined virtual file size (default 4 MB). By that less files are processed per spark partition. I/O and scanning time are not only becoming lower per partition but also the whole job speeds up. 4 MB is usually a good value. In some special situations and with loads of very small files you might want to adjust the OpenCostsInBytes.

Feel free to comment or challenge my explanations as always. Happy to learn also myself more by the community.

You want to master Data Engineering with PySpark? You want to see more videos like this? Subscribe here: https://www.youtube.com/@DataNikktheG...

Link to the code can be found here: https://github.com/datanikkthegreek/S...

00:00 - Intro to MaxCostsInBytes
02:55 - The dataset
03:25 - The results


#spark #pyspark #dataengineering #dataengineeringessentials