Vineyard, an open-source immutable in-memory data-manager that is designed for optimize the data exchange between tasks inside a bigdata analytical workflow, now supports airflow and can serve as the data backend for efficient data sharing between tasks.

See how vineyard helps for ETL jobs that large volume of complex data need to be shared between tasks: https://registry.astronomer.io/dags/v6d-etl-pandas

There's also a tutorial to get started: https://v6d.io/notes/airflow.html