- The homepage makes a vague statement about "AI". This is a personal thing, but to me "AI" sets off warning bells. It feels like a vague term, compared to "ML".
- I couldn't find any FAQ or references on the site to technical architecture. What underlying framework/library is used for the models, and what kind of NN architecture?
- If I were going to do this myself, I would probably do something like the below. Why not do this?
> Dump tabular data of feature/option/scores and model serialized as binary in SQLite DB
> Use a tool to autogenerate a CRUD API over SQL DB, host that as model server
> Use Litestream to replicate SQLite DB to S3
> Pick one of the beginner-friendly ML toolkits like PyTorch/Fast.ai/TF, use tabular data API to train prediction/scoring model from historical data
> Option to fetch the SQLite DB to local devices so that it's offline-capable
Also as a note, the quickstart guide redirects to iOS SDK Github
The reason to use a full machine learning model instead of a simple statistical model is three-fold:
1) The framework supports any arbitrary JSON encodable data structure as a variant, this includes nested dictionaries, lists, etc. A full ML model allows capturing the entire complexity of the data stored in the variant - numbers, strings, booleans, etc and those can even be changed and the model will still be able to predict performance.
2) A full ML model allows generalization across complex variants. So if later on you introduce new variants, it may already be imply some things about how that variant will perform. For the most part simple statistical models must learn each variant from scratch.
3) The models don't just seek to make decisions that are a global optimum, like you would do with A/B testing. They are seeking to find a contextual optimum given the current conditions. This type of modeling also requires the generalization that ML gives you.
Not sure I understand point #3 -- not familiar with global vs contextual optimum. The idea is that a global optimum is the aggregate optimum across all users, versus a contextual optimum which is the optimum for just the current user/device?
Couldn't you do this by doing regressions on a per-user-id basis or similar if that was the case? (I really don't know)