EventReduce: An algorithm to optimize database queries that run multiple times

Re:EventReduce: An algorithm to optimize database queries th... by danbruc on Thursday April 16, 2020 @08:33AM (#22888697)

Did I understand this correctly? You have a single set of items which you query by evaluating a predicate on each of them and then sort the matching ones. After the initial query you update the query result by looking at all the data update events, i.e. you remove delete items from the result, you insert matching new items in the correct position according to the sort order and you insert, remove, or move updated items as they start or stop matching the predicate and change their position according to the sort order.

[ Parent | on HN ]

Re:EventReduce: An algorithm to optimize database queries th... by eventreduce on Thursday April 16, 2020 @08:36AM (#22888713)

Yes this is correct. The performance benefit comes from doing all this stuff on the CPU instead of using disc-io. Also the internal binary decision diagram of EventReduced is optimized in a way to run less logic then a query would do. This makes it even faster then running the query again with an in-memory database.

[ Parent | on HN ]

Re:EventReduce: An algorithm to optimize database queries th... by ComodoHacker on Thursday April 16, 2020 @10:27AM (#22889689)

And the main cost of this (questionable IMO) benefit is losing consistency, which is losing any change to DB not coming from the calling app. You haven't mentioned this cost anywhere.

[ Parent | on HN ]

Re:EventReduce: An algorithm to optimize database queries th... by eventreduce on Thursday April 16, 2020 @10:47AM (#22889893)

The writes are not tunneled somehow through this algorithm. You still use the database like your normally would do. So the consistency is not affected.

Also this is an open source project, not something I want to sell you. Feel free to make a PR/issue with any open topics that are not mentioned in the docs.

[ Parent | on HN ]

Re:EventReduce: An algorithm to optimize database queries th... by ComodoHacker on Thursday April 16, 2020 @11:29AM (#22890399)

>The writes are not tunneled somehow through this algorithm

Then I fail to understand how it works. How Event-Reduce becomes aware of these "write events"?

>this is an open source project, not something I want to sell you

You made it open source so others can use it, right? They better be making an informed decision whether your solution suits their needs.

[ Parent | on HN ]

Re:EventReduce: An algorithm to optimize database queries th... by eventreduce on Thursday April 16, 2020 @11:34AM (#22890448)

You have to provide the events by yourself. See EventReduce as a simple function that can do oldResults+Event=newResults.

And yes, you should always do testings before you use open source stuff. There is no warranty use it on your own risk.

[ Parent | on HN ]

Re:EventReduce: An algorithm to optimize database queries th... by ComodoHacker on Thursday April 16, 2020 @11:54AM (#22890666)

OK, so you don't "tunnel writes through" EventReduce, you "tee" them to EventReduce.

Anyway, to maintain consistency, you have to limit yourself to one process of your app. No sharding, load-balancing etc. This is significant limitation, and it's not obvious. I encourage you to mention it in README.md.

[ Parent | on HN ]

Re:EventReduce: An algorithm to optimize database queries th... by eventreduce on Thursday April 16, 2020 @12:11PM (#22890826)

I encourage you to read the readme and check out the demo. EventReduce is nothing magically drills out your database and affects the consistency of your write-accesses.

It is a simple algorithm that is implemented as a function with two inputs and one output.

[ Parent | on HN ]

Re:EventReduce: An algorithm to optimize database queries th... by EdwardDiego on Thursday April 16, 2020 @06:32PM (#22894246)

> How Event-Reduce becomes aware of these "write events"?

Some DBs expose an event stream, for example, PG:

https://www.postgresql.org/docs/current/logicaldecoding-expl...

[ Parent | on HN ]

Re:EventReduce: An algorithm to optimize database queries th... by twic on Thursday April 16, 2020 @11:11AM (#22890181)

> For the different implementations in common browser databases, we can observe an up to 12 times faster displaying of new query results after a write occurred.

Is this intended to be an optimisation on top of localStorage and so on? If so, at least you don't have to worry about multiple writers.

[ Parent | on HN ]

Re:EventReduce: An algorithm to optimize database queries th... by eventreduce on Thursday April 16, 2020 @11:35AM (#22890455)

localStorage is no database, check out the demo page.

[ Parent | on HN ]