prostodata

Most difficulties in data processing arise from the need to process multiple tables. It is highly difficult in SQL but it can be also difficult in pandas especially for complex analytical queries. Indeed, in pandas you still have to use the same join and groupby operations as in SQL.

An alternative to SQL, join-groupby, and map-reduce is developed in the Prosto library:

- https://github.com/prostodata/prosto Functions matter! No join-groupby, No map-reduce.

It is a layer over pandas and its main distinguishing feature is that it relies on functions and function operations for data processing as opposed to using only sets and set operations in SQL and other set-oriented approaches

    comp = salary / (age - 18)

A better way to formally describe such calculated columns is to introduce functions and treat them as first class elements of the data model. In particular, function operations are better than set operations in these cases [1]:

- Calculating data using calculated columns. We describe a calculated column as a function without generating new relations (as opposed to select-from)

- Aggregating data. We can add an aggregate column without generating new relations (as opposed to groupby)

- Linking data. We can add derived link columns without generating new relations (as opposed to join)

This function-based approach to data modeling and data processing was implemented in Prosto [2] which demonstrates how many typical relational tasks can be solved using functions.

[1] Why functions and column-orientation? https://prosto.readthedocs.io/en/latest/text/why.html

[2] Functions matter! No join-groupby, No map-reduce. https://github.com/prostodata/prosto

SQL’s shortcomings can be grouped into these categories:

- lack of proper orthogonality — SQL is hard to compose;

- lack of compactness — SQL is a large language;

- lack of consistency — SQL is inconsistent in syntax and semantics;

- poor system cohesion — SQL does not integrate well enough with application languages and protocols.

[1] We Can Do Better Than SQL: https://www.edgedb.com/blog/we-can-do-better-than-sql/

https://github.com/prostodata/prosto - Functions matter! No join-groupby, No map-reduce

https://github.com/prostodata/prosto - Functions matter!

o Unifying ML API and patterns so that time series can be analyzed in the same way as normal tabular data. Example: sktime

o Preprocessing libraries applying data transformations to the input time series by converting them to a (tabular) form which can be used by normal ML algorithms. These are typically general or specific libraries for feature engineering, feature learning, feature selection and generic transformations. Examples: https://github.com/blue-yonder/tsfresh or https://github.com/prostodata/prosto

Which approach will win is not clear.

So the question is whether sktime can apply ARIMA in this way.

What is beautiful and consistent is the relational algebra. The relation model relies on this formalism to model data by making some rather strong assumptions about how tuples and relations represent things and how relational operations are used to process them. And these assumptions are precisely what propagates to SQL and what some authors (see references in the article) consider controversial and messy. Then the question is whether and how these controversies can be fixed (or whether they are bugs or features).

A radically different approach to fix these controversies is to introduce a different formalism and different data model (as opposed to fixing only syntax) which is based on using functions. In other words, instead of using sets and set operations, we use in addition functions and function operations [1]. Here you can find an implementation of this approach:

https://github.com/prostodata/prosto Prosto is a data processing toolkit radically changing how data is processed by using both sets and functions and being a major alternative to map-reduce, join-groupby and other set-oriented approaches

[1] Concept-oriented model: Modeling and processing data using functions: https://www.researchgate.net/publication/337336089_Concept-o... -> Read introduction (two pages) for why having only sets is not enough and why functions are important

One way to process data without joins and groupy in multiple tables is to use Prosto toolkit: https://github.com/prostodata/prosto It is an alternative to SQL, MapReduce and other classical approaches by radically changing how data is processed

Event driven concurrent framework for Python https://github.com/quantmind/pulsar

https://github.com/hardbyte/python-can

I used it (as well as Cantools) for importing various CAN-specific data formats.

It seems that in both cases the idea is to distribute (supposedly independent) requests between workers and one of the main difficulties is that requests might not be independent either within one stream (say, in the case of sessions) or between different streams (say, if they need to use one common state).