with s3+sqs+lambda+pandas - and you can build cheap serverless data processing pipelines and iterate extremely quickly
I think of lambda as a thread, and you can put a trigger on S3 bucket on each incoming file - to get processed. This allows you to get around GIL, and lets you invoke your lambda for each mini-batch.
assuming you have high volume and frequency of data - you will need to "cool down" your high frequency data, and switch from row-basis (like millions of rows per second) to mini-batch basis (like one batch file per 100Mb).
This can be achieved by having kafka with high partition number on the ingestion side, and sink to s3.
from S3 for each new file your lambda will be invoked and minibatch will be processed by your python code, and you can right size your lambda's RAM, but usually I reserve 2-3x size of a batch file for lambda.
the killer feature is zero ops. Just by tuning your minibatch size you can regulate how many times your lambda will be invoked
See a benchmark of Base R vs R + data.table (plus various other data wrangling solutions, including our own Easy Data Transform) at:
https://www.easydatatransform.com/data_wrangling_etl_tools.h...
You are aware the pandas was designed to replicate the behavior of base R's dataframes?
I've been a heavy user of both and R's data frames are still superior to pandas even without the tidyverse.
Pandas is really nice for the use case it was designed for: working with financial data. This is a big part of why Pandas's indices feel so weird for everything else, but if your index is a time in a financial time series then all of a sudden Pandas makes sense and works great
When not working with financial data I try to limit the amount of time my code touches pandas, and increasingly find numpy + regular python works better and is easier to build out larger software with. It also makes it much easier to port your code into another language for use in production (i.e. it's quick and easy to map standard python to language X, but not so much a large amount of non-trivial pandas).