Re:Building a new vector based storage model
by Darkphibre on Tuesday May 11, 2021 @05:04PM
(#27123498)
Oh, this is fascinating. Seven years ago I architected a true-realtime telemetry pipeline with end-to-end sequential guarantees (with roundtrip times <200ms excluding network latencies, and cloud processing times <20ms, leveraging BOND/ProtocolBuffer over AMQP over Websocket). It's still used by every 1st-party game for a large publisher.
It allowed for non-windowed event sequence analytics, enabling realtime feedback (think achievements that have multiple conditions).
And then the requirement was dropped, and (as you've found), everyone just uses it like a standard telemetry stream and is OK with 5-15min bins. :P
I still have a passion for the space, will definitely be reading up on this. I firmly believe this is the future of telemetry analytics; Congratulations on your efforts seeing the light of day!!
Disclaimer, I currently work for Microsoft, all words here are my own and do not necessarily reflect those of my employer, etc. ;)
The big decision was which direction to take to tackle the problem. LSM trees seemed an obvious choice, but we chose an alternative route so we wouldn't lose the performance we spent years building. Our latest release supports out-of-order ingestion by re-ordering data on the fly. That's what this article is about.
Also, we had many people asking about the differences between QuestDB and other open-source databases and why users should consider giving it a try instead of other systems. When we launched on HN, readers showed a lot of interest in side-by-side comparisons to other databases on the market. One suggestion [3] that we thought would be great to try out was to benchmark ingestion and query speeds using the Time Series Benchmark Suite (TSBS) [4] developed by TimescaleDB. We're super excited to share the results in the article.
[1] https://news.ycombinator.com/item?id=23975807
[2] https://news.ycombinator.com/item?id=23616878
[3] https://news.ycombinator.com/item?id=23977183
[4] https://github.com/timescale/tsbs