News for hackers. Stuff that matters.
Re:An opinionated map of incremental and streaming systems by PaulHoule on Tuesday May 04, 2021 @12:42PM (#27039564)
To me the unstructured area is the future of applications development and the structured area is barren and overhyped.

(e.g. 'eventually consistent' analytics is so 2021, because it looks like you are doing something quantitative,except it doesn't matter if you get the right answer. It will get you claps from a certain audience but most people lose patience pretty quick when the 'numbers don't add up'.)

Re:An opinionated map of incremental and streaming systems by dan-robertson on Tuesday May 04, 2021 @01:04PM (#27039825)
I can’t really relate your first paragraph to your second, at least by the taxonomy in the article which has some consistent structured systems as well as inconsistent systems.

I also disagree that unstructured is the future. I think most computations actually are structured. Otherwise sql queries wouldn’t be so useful. I think a lot of processes basically start with a big bag of foos and end up with a big bag of roughly corresponding bars, so if one foo changes then there is only a localised change to the output. That can be encoded with an unstructured system but I find they are better for cases where outputs are more scalar and the processes between inputs and outputs are myriad. While you can do structured-like things in an unstructured system, you lose out on a lot of the advantages of that structure.

Re:An opinionated map of incremental and streaming systems by PaulHoule on Tuesday May 04, 2021 @01:17PM (#27039952)
What I want is recognition of the structured and the unstructured.

For instance a tool like this

https://www.drools.org/

maintains a database of timely information and can use rules to match events and could be used to manage the numerous problems of event-based architectures.

That particular tool mangles Java source code badly in the process of compiling rules so it gives error messages that make no sense at all, even if you are looking at the compiler's source code in the other window and at the running compiler in the debugger.

In the light of the interest in "low code" and the real success of "business rules" for domains they apply in I am amazed there has been less effort to apply production rules to the "update the ui when the database changes".

Even though that can look unstructured, the implementation of the rules can be done with the same relational operators that all those other methods used. A few year ago you would have had to specified the indexes by hands, but the self-optimizing database patent from Salesforce is run out now and there is no reason the system can't learn the frequent query patterns itself.

Re:An opinionated map of incremental and streaming systems by elric on Tuesday May 04, 2021 @12:58PM (#27039755)
IMO the key with eventual consistency is making it known which bits are currently known to be consistent, and which are still in flux. I realize that this is vague, but how that works depends on what it is you're processing and how much time it takes for consistency to emerge.

If you're doing any kind of Important Reporting on eventually consistent data, you'd better make sure that you either know you're only including finalised data, or that there's a big fat warning with error bars.

Re:An opinionated map of incremental and streaming systems by jamii on Tuesday May 04, 2021 @07:11PM (#27044000)
This is why the high temporal locality part of the map is all EC - when everything is windowed you can just wait for the window to close.

On the low temporal locality side, the only consistent system I've seen so far is differential dataflow (and materialize) which does internally track which results are consistent and gives you the option to wait for consistent results in the output.

Re:An opinionated map of incremental and streaming systems by staticassertion on Tuesday May 04, 2021 @03:27PM (#27041824)
Exactly. People act like EC is impossible to make viable, and ignore the fact that transactional logic can impose 10-100x performance loss. Putting an EC system in front of a transactional system can be a massive performance win.

This isn't quite what the Helios paper talks about, but there's lots of things like async indexing in there that are kinda similar in nature