rockeetterark

database are often used to store data and search data by 'conditions', ex: `select * from ... where <conditions>`.

on the other side, in many cases we need to store MANY 'conditions' and search the stored conditions by 'data'. such as:

* online AD, the Advertisers define many conditions for user profiles, contexts ... * content filtering(forbidden words with very complex boolean expression) * risk control, alert ... conditions * data cleaning, curation... * auto labeling...

Such businesses often use Elastic Search percolator(or its underlying lucene monitor), more recent tantivy percolator(lucene's rust alternative).

Now, we have developed RuleDB for such use case, our enterprise users showing RuleDB is more than 1000x faster than ES(excluding RPC network overhead).

RuleDB has two parts: 1. the compiler and 2. the runtime.

1. The Compiler compiles rule code written by RuleDB DSL into binaries

  * AC automata binaries: for normal 'words'

  * **Multi** Regex engine: our **extended regex** with regular language algebras(not/and/or/concat/non-greedy-op...), many different extended regex in many rules are compiled into one DFA(the supported regex num  is 100x of hyperscan)

  * cost based recall optimization: for example in `A and B and C`, the least frequent of A,B,C is selected as the recall term, this is in recursive approach.

  * The rule verification VM: AC automata and regex just scan text for atoms, such atoms are combined with boolean expressions, ex: a near/3 quick near/4 "brown fox" and not(lazy near/+2 dog or diligent near/2 wolf).
     * The VM code are highly optimized by the compiler

2. The applications use RuleDB API calling the runtime lib

  * load the compiled binaries(just mmap)
  * scan + verify

Advanct features:

  * numbers:  integeral range `{i{3000,5000}}`, real number range `{r{2.71828,3.14159265}}`, -- save and matched as text, no width limit

  * composite index: `gender[1] and age[23] and income{i{3000,5000}}` will be compiled into an composite index 'i-gender-age-income'.

  * multi dimentions: the typical case geofencing: `longitude{r{116.2418,116.2441}} and latitude {r{39.5424,39.5450}}` -- RuleDB natively support any dimention search

  * such numeric expression and text expressions can be mixed: `(america or china) and gender[1] and age[23] and income{i{3000,5000}}`

A complex rule:

``` gender{{1}} and age{i{20,28}} and income{i{18000,23000}} and longitude{r{116.2418,116.2441}} and latitude {r{39.5424,39.5450}} and interesting{{movie|food|sport}} and books{{The Red and the Black|The Great Gatsby}} and a near/3 quick near/4 "brown fox" and not(lazy near/+2 dog or diligent near/2 wolf) ```

There are real world single rule with more than 50KB in our enterprise users, a rule source code file with 2MB is fully compiled in 150 milliseconds, a query using a 10KB document to search 70000 rules with 20 matched rules takes just 300us(micro seconds, not milli seconds)!

---- I'am also the author of ToplingDB, I had posted several months ago https://news.ycombinator.com/item?id=44432322

Thank you for highlighting our project's activity! You're absolutely right -- this project is actively developed and growing .

Thank you, we have fixed READM, now it's very easy to start with ToplingDB by the tutorial in README.

Yes, we have S3 solution with ToplingDB for enterprise users.

Thank you!

Sure, MyTopling(MySQL) is based on ToplingDB and get the perf gains. There are other DBs also based on ToplingDB.

We keep the core component as private repo, which builds the ToplingZipTable files -- the SST with the searchable compression algo for speed. -- Although even without this component, ToplingDB is still outperforms RocksDB.

Sure, this is one of the reason we develop the framework of json/yaml conf and the embedded http server, with this framework, we discover several RocksDB bugs by this framework(we have contributed 100+ PRs and 60+ issues for RocksDB).

With embedded http server, all DB configurations and status are visually displayed, for example: the compaction progress, the SST file list with the compacting files are highlighted -- with different colors for different compaction jobs, different icons for different compaction types(https://github.com/topling/sideplugin-wiki-en/wiki/Compactio...)

It depends on workloads, in most cases, there is at least 3x perf gain. One of our customer get 20x perf gain(point search).

Another significant improvement is `select count(*) from SomeTable` in MyTopling, I optimized such query by 30x+ faster than MyRocks, 10x+ faster than InnoDB, while the multi-thread scaling is near linear -- better than InnoDB.

Yes, you have exactly got the key points.

Yes, ToplingDB embeds an http server for showing DB info and online conf changes, this http server is very light weight and does not introduce any visible overhead.

-- prometheus is supported by this this http server.

Todis is a massive, persistent Redis server developed by Topling Inc which is a new start up company.

The Todis author was also the author of TerarkDB/Terark which was acquired by Bytedance at 2019.

Todis is Cloud Native, users can experiencing Managed Todis based on aliyun in 10 minutes.

The key features of todis are:

   1. Computing and Storage are seperated for elastic scaling
   2. Auto scaling, no need for sharding/partitioning
   3. Elastic Distributed Compaction(feature of ToplingDB which forked from RocksDB)
   4. Plenty (grafana) monitoring metrics
   5. Web view for DB internal stats

The day they offer a download, I promise you we'll run a benchmark ;)

Sure! Our benchmark against MySQL is here: https://github.com/Terark/mysql-on-terarkdb/wiki/YCSB-on-9.1... We used YCSB on 9.1Gb of movie data. This benchmark is comparing MySQL with our product "MySQL on Terark". "MySQL on Terark" is basically MySQL configured with TerarkDB instead of InnoDB -- that way you can migrate your MySQL applications to Terark with virtually no modification in your code.

We do not have any benchmark against PostgreSQL though. It is not in our plans to adapt our storage engine to PostgreSQL, so we're not comparing it against it, but the gains are just as significative.

I hope that answer your questions, and feel free to reach us at [email protected]

Yep, this is an article we published on our blog to give a bit of background information on databases performance. It's very general.

For benchmarks on TerarkDB's performance in particular, you can have a look here: https://github.com/Terark/terarkdb/wiki/Benchmark

Thanks for mentioning it! Yes, for Chinese speakers, our CTO Lei Peng has answered a lot of questions on Zhihu (the Chinese Quora).

It works, but it's not the best scenario for us. Scenarios with financial data are most likely sequential read (gimme all data for the last 50 trading days) and write heavy (tick-level write). We're blowing away the rest of the pack when you have a huge haystack and you're looking for the needle in it, that's where you're gonna get a 200x boost in performance with using Terark.

In our situation, making it fully open source for wide adoption would oblige us to become a consulting company with revenue based on support. This would require a large headcount that we cannot support as a mostly bootstrapped startup. So, for now, we decided to focus on the tech and deliver the best tech possible to the (largest) clients who need it the most. Think of it as Quality vs/ Quantity. For instance, as reported in the article, our largest client at the moment is Alibaba Cloud (the Chinese Amazon AWS). We're able to cater to their custom needs and even send our CTO and some engineers to their office to accompany them when needed. We solve a pain point for them on a huge scale, and we're able to make a decent revenue that makes us profitable and allows us to grow our business independently.

But we're open on the question. Our storage engine is also compatible with MongoDB and MySQL, so if we could partner with a large company providing support for MongoDB/MySQL on Terark (think something like Percona, for instance), and that open sourcing all our code was a must, we would consider it.

Thanks. These assertions are here to give a basic background and overview of databases performance in general. The real game changer with Terark is our novel compression algorithm. It's more space efficient, that's one thing, but above all else we can search directly into the compressed data without decompressing it. That's the real breakthrough.

We do that by using a data structure called Succinct Nested Trie, and we've introduced concepts such as CO-Index (Compressed Ordered Index) and PA-Zip (Point Accessible Zip).

We were at first a compression company, and turned to storage engines and database as a domain of application for our algos, hence the analogy with Pied Piper :)

Thank you, we really appreciate your comment. As I mentioned in an other reply, we understand the claim may sound outlandish, so we try to be as transparent as possible:

- We provide several different benchmark results and detailed procedures: https://github.com/Terark/terarkdb/wiki/Benchmark - We provide a free license of TerarkDB and you can download the exact scripts we used and run your own benchmarks with the configuration you want.

We're a bunch of geeks (I think the picture is worth a thousand words ^^) who had a scratch to itch and a lightbulb moment. We built a product around it and we're trying to make a sustainable business. Any feedback or comment is welcome. We understand some people might be skeptical and we're happy to answer any question. And if you like it, we would be thrilled if you could help spread the word. We're not the bests at marketing... haha

That is correct. We simply built a storage engine technology. You run it yourself on your own servers. Everything is open source except for the core compression algorithms that are loaded as a proprietary dynamic library. Being a young startup, we made the choice not to open source this part for now (see my other comment higher in this thread).

In 200x improvement in random read performance, compared to RocksDB or WiredTiger (MongoDB's storage engine). You can find benchmarks on our website: https://terark.com/en/index And here: https://github.com/Terark/terarkdb/wiki/Benchmark

We also provide a free license of TerarkDB and you can download the exact scripts we used and run your own benchmarks with the configuration you want. We know the claim may sound outlandish, so we try to be as transparent as possible.

TerichDB is an experimental repo. We'll take it private to avoid confusion. Thanks for pointing this out.

Regarding the license of our products: the core of TerarkDB is a plug-in for RocksDB. It is loaded as a dynamic library for librocksdb.so and compliant with RocksDB’s license. All the code related to MongoDB/MySQL is open source (We use MongoRocks and MyRocks).

Making the core algorithms open source is a dilemma for us. At this stage, as a young startup, keeping the core algorithms proprietary gives us leverage on the valuation and insulates us from potential competitors. But this is something we may reconsider in the future in order to facilitate a wider adoption of our products.

It's a big debate, even for us internally. If anyone else here has been facing the same dilemma, we'd love to hear about your opinion, what you chose in the end and how things turned out.

Sorry, I know the article is kind of sensational but it also has some good information and we're here to discuss the real substance in the thread.

Terark built a new storage engine for Database and Data Systems based on the Succinct Nested Trie data structure. Our technology enables direct search on highly compressed data without decompressing it. Thanks to that we obtain >200X faster performance and more than 15X storage savings (better than Google's LevelDB or Facebook's RocksDB). We are a Y Combinator company (W17).