There's something wrong with the Pandas API on Spark
Fix the following issues: Errors converting large datasets to pandas, pandas for Spark is very slow, and pandas for Spark column reduction doesn't reduce data.
https://endjin.com/blog/2024/1....1/theres-something-w
Discover the world at Altruu, The Discovery Engine