asavinov

https://github.com/asavinov/intelligent-trading-bot

It trains ML models based on historic data and custom features and then uses them to generate a kind of intelligent indicator between -1 and +1. This intelligent indicator is then used to make trade decisions. Frequency is a parameter and can very from 1 minute for crypto trading to 1 day for normal exchanges.

    FROM Table AS t WHERE t.Condition SELECT t.col1, t.col2, ...

    SELECT t.col1, t.col2, ... FROM Table AS t WHERE t.Condition

    Loop -> SELECT-FROM-WHERE
    Table -> Collection
    AS t -> Loop instance variable
    WHERE -> condition on instances

    foreach x in Collection
        if x.field == value:
            continue
        // Do something with x, for example, return in a result set

In Python, loops also specify the collection first:

    for x in Collection:

    [(x.col1, x.col2) for x in Collection if x.field2 == value]

Yet, there exist problems where even old events retain their strength. An example is where we want to take into account discrete events (tokens in LLM) for predicting stock prices. These events might be explicitly defined (holidays, company announcements, important economic figures etc.) or derived from the data like technical patterns. The strength of transformers are in their ability to ignore the order of events and ignore the distances between them. More precisely, transformers can learn when it is important. In language models, this is used to generate output sequences where semantically equal tokens have completely different order than in the input sequence. Something similar can be done in time series forecasting if we accordingly define "tokens", for example, as technical patterns. Then rising stock prices can be explained (and predicted) not only because of recent numeric behavior but also because "something happened" two weeks ago.

[0] https://github.com/asavinov/intelligent-trading-bot Intelligent Trading Bot: Automatically generating signals and trading based on machine learning and feature engineering

[0] https://github.com/asavinov/intelligent-trading-bot Intelligent Trading Bot: Automatically generating signals and trading based on machine learning and feature engineering

If you process many positions and then choose top 5 candidates then do you choose one of them for buying or you can allocate resources between them (depending on some score)?

- Data feeds and data ingestion. It can be a fairly independent component which collects data from different sources (might be even discussion forums) making it available to other components in some uniform format

- Feature generation. The source data is rarely used in its original form for decision making and having good (informative) features is frequently the primary factor of success. Moving averages is an example but nowadays this will hardly work

- Signal generation. Here some logic should be applied in order to emit discrete decisions and such models are heavily parameterized with thresholds.

- Real trading and order management as well as coordination of all activities.

The article sheds some light on the technological aspects and the general pipeline used to process the data and manage orders. Although it might be interesting indeed, I would expect more details about how to scale the solution and how to implement it asynchronously. Especially if it uses Go which has a special construct for that purpose - channels.

I understand that it is not the focus of the article, but having some general information about its trading logic and how to plug new and parameterize existing strategies would help. Some links at the end are quite interesting for me because I am developing an intelligent trading bot based on ML and feature engineering (https://github.com/asavinov/intelligent-trading-bot) for which such articles might be quite important

At the end it is necessary to make a decision whether to buy or sell (and how much), which will compete with other decisions made based on some logic. Developing such a logic (strategy) manually is of course quite difficult. I developed an intelligent trading bot which derives its trading strategy from historic data:

https://github.com/asavinov/intelligent-trading-bot

Currently it works for cryptocurrencies but can be applied to other markets:

https://t.me/intelligent_trading_signals

> I've spent months on figuring out the best parameters for trading. Ended up this working only on historical data, while in reality it was totally different.

It is a typical situation. The whole problem is to develop a strategy which works for future (unseen) data. Even backtesting algorithms should be designed in such a way that future experience (data) does not leak to the past.

- How connections between multiple tables are represented and managed

- How derived data is described (queries, workflows etc.)

Typically such tools are aimed at simplifying data connections but normally they end up with some kind of join-like approach which requires high expertise and is error prone. So users have to deal with something they wanted to get rid of when they buy the tool. Plato is not an exception: "No SQL needed". Yet, I could not find any information on how exactly it manages connections between tables and how the unified "virtual table" is defined.

The second question is about how we can derive new data from existing data. Ideally, users would like to have something very similar to Excel because spreadsheets are indeed extremely intuitive: we define new cells as functions of other cells (which in turn might be functions of other cells). In Plato I found "virtual columns" which should be rather useful. This is somewhat similar to the column-oriented approach implemented in Prosto [0]. Yet, what is really non-trivial is how to define (derived) columns by combining data from multiple tables.

In general, the tool looks very promising and I hope that additional features and additional information will make it really popular.

[0] https://github.com/asavinov/prosto Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby

I also developed an ML-powered service heavily relying on feature engineering

https://github.com/asavinov/intelligent-trading-bot Intelligent Trading Bot

Its difference from Didact is that this intelligent trading bot is focused on trade signal generation with higher frequency of evaluation. It is more suitable for cryptocurrencies but also works for traditional stocks with daily frequencies so it could be adapted for stock picking. What I find interesting in your work is the general design of such kind of ML systems relying on feature engineering.

It is the central part of RM which is difficult to model using other methods and which requires high expertise in non-trivial use cases. One alternative to how multiple tables can be analyzed without joins is proposed in the concept-oriented model [1] which relies on two equal modeling constructs: sets (like RM) and functions. In particular, it is implemented in the Prosto data processing toolkit [2] and its Column-SQL language [3]. The idea is that links between tables are used instead of joins. A link is formally a function from one set to another set.

[1] Joins vs. Links or Relational Join Considered Harmful https://www.researchgate.net/publication/301764816_Joins_vs_...

[2] https://github.com/asavinov/prosto data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby

[3] Column-SQL https://prosto.readthedocs.io/en/latest/text/column-sql.html

This approach was formalized in the concept-oriented model of data which relies on two basic elements: mathematical functions and mathematical sets. In contrast, most traditional data models rely on only sets. Functions are implemented as columns. This model gets rid of joins and groupby by making data processing simpler and more intuitive.

This approach was implemented in the Prosto data processing toolkit: https://github.com/asavinov/prosto

A similar approach is implemented in the Prosto data processing toolkit:

https://github.com/asavinov/prosto

Although Prosto is viewed as an alternative to Map-Reduce by relying on functions, it also supports Python User-Defined Functions in its Column-SQL:

  prosto.column_sql(
    "CALCULATE  Sales(quantity, price) -> amount",
    lambda x: x["quantity"] * x["price"]
  )

- https://rows.com/

- https://www.rowshare.com/

- https://www.airtable.com/

- fieldbook

- Power BI

  ColumnA = ColumnB + ColumnC, ColumnD = ColumnA * ColumnE 

Yet, the main problem is in working multiple tables: how can we define a column in one table in terms of columns in other tables? For example:

  Table1::ColumnA = FUNCTION(Table2::ColumnB, Table3::ColumnC)

Why it is difficult to define new columns in terms of other columns in other tables? Short answer is that working with columns is not the relational approach. The relational model is working with sets (rows of tables) and not with columns.

One generic approach to working with columns in multiple tables is provided in the concept-oriented model of data which treats mathematical functions as first-class elements of the model. Previously it was implemented in a data wrangling tool called Data Commander. But them I decided to implement this model in the Prosto data processing toolkit which is an alternative to map-reduce and SQL:

https://github.com/asavinov/prosto

It defines data transformations as operations with columns in multiple tables. Since we use mathematical functions, no joins and no groupby operations are needed and this significantly simplifies and makes more natural the task of data transformations.

Moreover, now it provides Column-SQL which makes it even easier to define new columns in terms of other columns:

https://github.com/asavinov/prosto/blob/master/notebooks/col...

https://github.com/asavinov/prosto

However, Prosto allows for data processing via column operations in many tables (implemented as pandas data frames) by providing a column-oriented equivalents for joins and groupby (hence it has no joins and no groupbys which are known to be quite difficult and require high expertise).

Prosto also provides Column-SQL which might be simpler and more natural in many use cases.

The whole approach is based on the concept-oriented model of data which makes functions first-class elements of the model as opposed to having only sets in the relational model.

One approach to radically simplify operations with data is to use mathematical functions (in addition to mathematical sets) which is implemented in Prosto data processing toolkit [0] and (new) Column-SQL [1].

[0] https://github.com/asavinov/prosto Prosto is a data processing toolkit - an alternative to map-reduce and join-groupby

[1] https://prosto.readthedocs.io/en/latest/text/column-sql.html Column-SQL (work in progress)

More details about "Why functions and column-orientation" (as opposed to sets) can be found in [2]. Shortly, problems with set-orientation and SQL are because producing sets is not what we frequently need - we need new columns and not new table. And hence applying set operations is a kind of workaround due the absence of column operations.

This approach is implemented in the Prosto data processing toolkit [0] and Column-SQL[1] is a syntactic way to define its operations.

[0] https://github.com/asavinov/prosto Prosto is a data processing toolkit - an alternative to map-reduce and join-groupby

[1] https://prosto.readthedocs.io/en/latest/text/column-sql.html Column-SQL (work in progress)

[2] https://prosto.readthedocs.io/en/latest/text/why.html Why functions and column-orientation?

https://github.com/asavinov/lambdo - Feature engineering and machine learning: together at last!

Yet, here the focus is on feature engineering and rethinking how it can be combined with traditional ML. Essentially, the point is that there no big differences and it is more natural and simpler to think of them as special cases of the same concept: features can be learned and ML models are frequently are used for producing intermediate results.

https://prosto.readthedocs.io/en/latest/text/why.html

Column is an implementation of a function (similarly to how table is an implementations of a set). Theoretically, this approach leads to a data model based on two core elements: mathematical functions (new) and mathematical sets (old).

This approach was implemented in Prosto which is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby.

The relational model is relies on set theory (more specifically relational algebra). An alternative view on data and data modeling is based on 1) sets, and 2) functions, and is called the concept-oriented model [1, 2]. It is actually quite similar to category theory and maybe even could be described in terms of category theory. It is also quite useful for data processing and there is one possible implementation which is an alternative to map-reduce and join-groupby approaches [3].

[1] A. Savinov, On the importance of functions in data modeling https://www.researchgate.net/publication/348079767_On_the_im...

[2] A. Savinov, Concept-oriented model: Modeling and processing data using functions https://www.researchgate.net/publication/337336089_Concept-o...

[3] https://github.com/asavinov/prosto Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby

One approach to modeling data based on mappings (mathematical functions) is the concept-oriented model [1] implemented in [2]. Its main feature is that it gets rid of joins, groupby and map-reduce by manipulating data using operations with functions (mappings).

> Everything is pre-joined — you don’t have to disassemble objects into normalised tables and reassemble them with joins.

One old related general idea is to assume the existence of universal relation. Such an approach is referred to as the universal relation model (URM) [3, 4].

[1] A. Savinov, Concept-oriented model: Modeling and processing data using functions, Eprint: arXiv:1911.07225 [cs.DB], 2019 https://www.researchgate.net/publication/337336089_Concept-o...

[2] https://github.com/asavinov/prosto Prosto Data Processing Toolkit: No join-groupby, No map-reduce

[3] https://en.wikipedia.org/wiki/Universal_relation_assumption

[4] R. Fagin, A.O. Mendelzon and J.D. Ullman, A Simplified Universal Relation Assumption and Its Properties. ACM Trans. Database Syst., 7(3), 343-360 (1982).

    SELECT SUM(A) FROM MyTable

Something similar is implemented in these libraries which however rely on a different data processing conception (alternative to map-reduce):

https://github.com/asavinov/prosto - Functions matter! No join-groupby, No map-reduce.

https://github.com/asavinov/lambdo - Feature engineering and machine learning: together at last!

o Generally, we do not want to re-compute aggregates - aggregates should be also updated, particularly, if n is very large

o In real applications, rolling aggregation is performed using partitioning on some objects. For example, we append new events from many different devices to one table and want to compute rolling aggregates for each individual device. Hence, this (i-n, i+n) will not work anymore.

o Rolling aggregation using absolute time windows will also work differently. Although, if records are ordered (like in stream processing) and there are no partitions, then it is easy.

There are the following difficulties in implementing such systems:

o (Small) changes in input have to be incrementally propagated to the output as updates rather than new results. This changes the paradigm of data processing because now any new operator has to be "update-aware"

o Only simple operators can be easily implemented as "update-aware". For more complex operators like aggregation or rolling aggregations, it is frequently not clear how it can be done conceptually (efficiently)

o Differential updates have to be propagated through a graph of operations (topology) which makes the task more difficult.

o Currently popular data processing approaches (SQL or map-reduce) were not designed for such a scenario so some adaptation might be needed

Another system where such an approach was implemented, called incremental evaluation, is Lambdo:

https://github.com/asavinov/lambdo - Feature engineering and machine learning: together at last!

Yet, this Python library relies on a different novel data processing paradigm where operations are applied to columns. Mathematically, it uses two types of operations: set operations and functions operations, as opposed to traditional approaches based on only set operations.

A new implementation is here:

https://github.com/asavinov/prosto - Functions matter! No join-groupby, No map-reduce.

Yet, currently incremental evaluation is implemented only for simple operations (calculated columns).

Feature engineering in general is a hot topic and especially if features are not simple hard-coded transformations but rather can be learned from data. For example, I developed a toolkit intended for combining feature engineering and ML:

https://github.com/asavinov/lambdo - Feature engineering and machine learning: together at last!

(Currently, it is not actively developed and the focus is moved to a similar project - https://github.com/asavinov/prosto - also aimed at data preprocessing and feature engineering)