kootenpv

On phone I typed with a capital letter the correct word but it was considered wrong.

I have 2 popular python projects, one with 4.9k and one with 2.3k stars and I don't qualify :/

https://github.com/kootenpv

If anyone knows why pls let me know

Models are cached and not large so the setup time is very low. I'll time it when I get to a PC

Perfect interview questions to get to the juice details haha!

The problem is that choice of compression is very much dependent on the sample size, so this is why just choosing the algorithm based on running benchmarks on the sample will be off.

However, computing certain statistics on the sample and then doing machine learning makes a lot of sense!

Obviously not for simple/cheap to compute features such as number of rows/columns which actually do not take longer to compute as the data gets larger.

But I actually do this for predicting the uniqueness of values! You basically want to get an idea of how many unique values you have per column. But.. if you take a sample this will give you completely wrong numbers.

You can see my approach here: https://github.com/kootenpv/shrynk/blob/6a8675061d82aa65fc3b... Basically the formula calculates the uniqueness on a sample (e.g. 10000 rows), but then extrapolates the result to your actual data size. E.g. it finds 100 unique values in a sample of 10000, but in reality that means you maybe have 500 unique values.

Yea, I thought parameters per compression algorithm should indeed be added in a next version :) more compute but definitely an improvement. I think pandas doesn't offer zstd as option with csv, but I'll check once more.

EDIT: indeed, it's missing in `to_csv` - seems like an oversight.

Yea zstd is really amazing... if I would choose a single one all the time it'd be zstd for sure.

I can give you this... the larger the data the more useful parquet and compression on it will be...

Yea I am aware of meta hyperparameter approach for ML, except they only focus on accuracy instead of also including train/prediction times in to the equation :) That's what I was referring to! (you can save A LOT of compute and zoom in on things that work if you can weed out slow / badly performing algorithms as part of meta learning hyperparameters).

To make it extra clear: by doing a lot of compute on different datasets and not only recording the accuracy but also time it took, and then by including that as dimension it will even give better results.

Yea, look at jekyll in combination with github pages. You can see my blog for example (https://vks.ai), the code is hosted here: https://github.com/kootenpv/kootenpv.github.io

There's TimescaleDB built on top of PostgreSQL.

Indeed, but this is for lossless compression :)

Spot on (I briefly touch on this in the article)! This is why I try to work with cheap-to-compute features. I used to calculate how unique all values were, but ended up taking a sample instead to speed that part up for large data!

Yea - that is related to genetic programming. That, and using auto-encoders for e.g. image compression are known approaches in "AI".

I'm particularly proud of this meta approach and I am actually thinking this could become huge: the same thing can be done for hyperparameter optimization in machine learning tasks.

Hyperparamter optimization is currently focused on minimizing cross-validation error, but using this concept you could have weights on accuracy, training time and prediction time (very similar to compression where the 3 dimensions are size, write time and read time), and then given a new unknown dataset you could predict what model/hyperparameters to use.

Maybe this should be patented ;)

I like mine for the fact that it is static and hosted on github pages with a not too bad design.

https://vks.ai/

shrynk is not doing any compression itself, it's using ML to do a "meta" approach: it computes features on your data like how many rows, how many columns, how much duplication etc, and then predicts which existing compression algorithm (out of the available compressions) will be best given your requirements/preferences in terms of size, write time and read time.

I'm a long time fan of your blog :O

Definitely! I really optimized for "no development time spent" and was just using pandas to extract html tables into csv and just store them :-). 2 lines of code really. I had no idea I would have it running for so long.

It was really just the example that made me wonder why I have to consider which compression would be best for files with my characteristics - but not saying it was best practice to begin with haha!

Privacy & Personal Analytics

I am working on an open-source implementation that is based on the fundamental idea of privacy. The consumers should have full control eventually.

For now only Google etc have all the information on us, but the great thing is that because of GDPR we see a move towards us being able to download it.

The system I am building allows us to combine data from across companies. This is huge. We are talking so much opportunity. The real challenge is making it easy for people to onboard while still providing security.

Feel free to contact me if you're more interested.

I'm happy to share with you the result of this project, it certainly has been a lot of fun!

First of all it was shocking how many messages we had over such a long time (more than 300k!)... and it was very interesting to see the patterns occurring. Next, I trained a bot for several nights (+ experimentation) that in the end does pretty well. From the video, you can see that it doesn't take the current time into account, which actually turned out to be its biggest flaw, lol (no spoilers)!

Anyway, writing the "facebook driver" with selenium was fun too. What turned out to be an interesting part of the model is that it had to split up a message into more messages (there's a difference between a newline and a new bubble), which I represented as a special character in my data. The other way too - I had to make sure it did not reply to every message individually, but instead, I had it wait some time. It was considering messages starting with a capital letter as "mobile" (and allow a longer wait than non-mobile).

Anyway, I hope you enjoy!

If you like it, please follow me on https://github.com/kootenpv or my blog https://vks.ai

The best article I have read in a long time.

Are you saying there is some library already existing? I'd be curious to try it then.

Yes, these things should ideally be taken into account. I have been contacted by a company that is actually working on it: http://sourced.tech/

Haha! That's gonna be scary.

Yea, I also have a similar project in the making: https://github.com/kootenpv/xtoy . This one does optimisation of finding a machine learning model using evolutionary search, but mainly focuses on just taking any kind of data and coming up with a prediction (missing data, text data, date/time data etc). It's a lot of fun :)

You are totally right: I was mixing up with another project I'm working on where I am using seq2seq (using only tensorflow) :) I will update the text of the repo. Thank you!