News for hackers. Stuff that matters.

A comment on Emerging Architectures for Modern Data Infrastructure. Parent comment

Re:Emerging Architectures for Modern Data Infrastructure by sradman on Sunday October 18, 2020 @06:17AM (#24816750)
> it ignores sensor data sources, which are both the highest velocity and highest volume data models by multiple orders of magnitude.

This has long been the main marketing message used to promote Complex Event Processing (CEP) [1] systems. There is no shortage of enterprise and Open Source solutions for this space; what is missing is strong demand/adoption which in itself undermines the next-big-thing claim.

One can argue that sensor data is included in the ETL category.

[1] https://en.m.wikipedia.org/wiki/Complex_event_processing

[ Parent | on HN ]
Re:Emerging Architectures for Modern Data Infrastructure by jandrewrogers on Sunday October 18, 2020 @04:10PM (#24820723)
For many sensor data models, CEP is a core element of the data flow but the constraint/query data model is much larger and more dynamic than is typically supported in classic off-the-shelf CEP systems.

This isn't necessarily an issue, complex constraint matching is typically a fundamental part of the ingest path anyway given the algorithms used; making it support more generalized CEP is a fairly straightforward extension of the same computer science mechanics that make polygon search scale efficiently.

[ Parent | on HN ]