So then the question is.. why have R/Python/etc. reinvented the wheel with their own datatable structures instead of using SQL?
It’s not really reinventing the wheel, I make use of sql, pandas, pyspark, sparksql, data.table, bigquery, it all depends on the use case.
Dataframe abstractions are really easy to understand so I have worked with journalists and biologists who are able to work productively with pandas or R. Transforming and analyzing data is often easier to write with pyspark while spark sql makes it easy to access and extract data from some data sources.
It all really depends on who is your customer, who are your coworkers, how much volume of data do you have, how much time do you have to accomplish your task etc.
Other reasons: every language can use a SQLite db, you can add indexes, you can add multiple tables in one file, you can store large amounts of data on disk, it has built in json queries, there’s a great ecosystem of SQLite libraries, support for it is built into python.