ehsantn

The article highlights important challenges regarding Python performance optimization, particularly due to its highly dynamic nature. However, a practical solution involves viewing Python fundamentally as a Domain Specific Language (DSL) framework, rather than purely as a general-purpose interpreted language. DSLs can effectively be compiled into highly efficient machine code.

Examples such as Numba JIT for numerical computation, Bodo JIT/dataframes for data processing, and PyTorch for deep learning demonstrate this clearly. Python’s flexible syntax enables creating complex objects and their operators such as array and dataframe operations, which these compilers efficiently transform into code approaching C++-level performance. DSL operator implementations can also leverage lower-level languages such as C++ or Rust when necessary. Another important aspect not addressed in the article is parallelism, which DSL compilers typically handle quite effectively.

Given that data science and AI are major use cases for Python, compilers like Numba, Bodo, and PyTorch illustrate how many performance-critical scenarios can already be effectively addressed. Investing further in DSL compilers presents a practical pathway to enhancing Python’s performance and scalability across numerous domains, without compromising developer usability and productivity.

Disclaimer: I have previously worked on Numba and Bodo JIT.

Bodo is a JIT compiler for Pandas/Numpy code that statically types the code during compilation time. However, it infers types automatically (looks at file metadata etc) and not really built for manually typing everything.

https://github.com/bodo-ai/Bodo (I work on Bodo)

I think staying in Python is best to ensure the team's productivity. There are a lot of tools that help accelerate and scale Pandas code while trying to be as compatible as possible. You can look at "Pandas on Spark", Snowpark (if on Snowflake), Bodo, Dask, and Modin for example.

Disclaimer: I work on Bodo and wanted to share it in case others find it useful. https://github.com/bodo-ai/Bodo

Thanks!

Dask's architecture is unable to handle large scale data processing tasks (e.g. ETL, dataframe operations) because it's just a distributed task scheduler. It works as long as tasks have very little communication across them. But data processing tasks like table join need heavy communication (e.g. shuffle) which requires a true parallel architecture like MPI to be done efficiently. Almost all of Dask's problems mentioned here go back this issue.

Bodo is a new compute engine that brings true parallel computing with MPI to data processing. Bodo is over 100x faster than Dask for large-scale data processing. Forget about speed, are you willing to pay AWS/Azure/GCP 100 times more than necessary (also increasing your carbon footprint)?

Bodo uses a new inferential JIT compiler technology which requires getting used to but it handles actual Pandas APIs (not "Pandas-like").

Disclaimer: I work for Bodo.