Disclaimer: I work at VictoriaMetrics open source.
VictoriaMetrics ingest rates are around 300k / per second / PER CORE. So theoretically you should be fine with just a single n1-standard-32 or *.8xlarge node. Though I would recommend cluster version for reliability, of course, and to scale storage/ingestion/querying independently.
Here's the benchmarks with charts: https://medium.com/@valyala/measuring-vertical-scalability-f...
As to what these companies want to do with sensor data, it is often considerably more interesting than what people imagine. Many of the applications have an operational real-time or low-latency tempo. (I can't be too specific here.)
For my purposes, I put "medium-sized" on the order of $1B annual revenue. As to why a company would literally spend 10+% of its revenue on sensor data infrastructure, it is difficult to overstate the extent to which getting this right is viewed as near- to medium-term existential for these companies. The CFO has run the models and this is their best chance at survival.
Here is the interesting thing: to the extent they've been able to put this sensor data infrastructure in place, it has been successful at generating margin. If they could bend the infrastructure cost curve down a bit, most would spend even more on it. I've seen the financial models at several companies, there is a tremendous amount of money to be made in this transformation.
Is this about optimizing things to a precision that is impossible when humans try to decide whether something is too much or too little of something?
In the short-term, big tech companies can't replicate what is possible for these hardware companies. Longer term, I would expect these data sources to be commoditized as well.
Let's take plastic injection molding since it's such a good example of a really broken industry (there are a small number of excessively competent injection molders and a vast legion of incompetent ones).
You're shooting a part every couple of seconds (or faster), and that injection molding machine has lots of knobs to dial in. Temperature of incoming plastic pellets, water content of incoming plastic pellets, dye feed rate, plastic feed rate, mixing chamber temperature, feed screw motor load, initial injection pressure, plateau injection pressure, release injection pressure, actual pressure inside the mold, time spent cooling--I can go on and on and on.
Most injection molding problems generally get solved one way: increase injection time. It's fairly straightforward to adjust, isn't likely to make things go wrong, and the people on the line don't get paid to experiment. They've got 100K parts to shoot in 72 hours, and an hour lost is a thousand or so parts they're going to get yelled at for. Better to dial the time up 10% and take 79 hours rather than experimenting for 7 hours and not shooting or waste a bunch of plastic.
Of course, if this is your only hammer, you can see where this is going. Every single time something goes wrong, that mold gets another 10% added to its cycle time. And it never goes the other way without "A Pronouncement From God, Himself(tm)". Eventually, your entire business is running at 50% productivity because all the molds are shooting so slow and you think you need to build another factory when what you need to do it fix your molding times.
Now, back to sensors--the problem is that nobody with incentive has a way to identify AT THE TIME IT CAN BE DEBUGGED that "something is going wrong". Someone on the line dialing up the injection time should cause an immediate dump of ALL the data on that machine (probably a week or more+) up to an engineer who can go through it looking for anomalies. Even better would be for the machine to flag to an engineer any "likely anomalies" (an increase in incoming plastic pellet water content should get flagged, for example) so that they can be corrected before they affect the injection process and cause failures/wastage.
This is, of course, all predicated on logging an enormous amount of data and being able to run an analysis against it in almost real-time.
Handling this data is non-trivial.
One of the things that struck me then is how much the line workers were treated like furniture, when many of them were quite sharp. They took such pride in getting things done well and at speed, in continually improving. I really wish I could put that kind of data in the hands of a couple of the people who trained me. Just an app on their phones. Spending 40+ hours/week on a machine means you really get to know it. I'd love to see how many of them would get great first-pass analysis and remediation.
Honestly from working in plastics and sensor design for 15 years it usually boils down to engineers not willing to let go of information because they envisage potential future issues. Its easier to imagine problems in a meeting than to imagine and deliver solutions upfront before the problem ever happens. Also, a lack of care for the economics of doing such sampling.
That's not to say there is an easy fix. These same people are the ultimate end customer who have the final word on such engineering environments.
This has long been the main marketing message used to promote Complex Event Processing (CEP) [1] systems. There is no shortage of enterprise and Open Source solutions for this space; what is missing is strong demand/adoption which in itself undermines the next-big-thing claim.
One can argue that sensor data is included in the ETL category.
[1] https://en.m.wikipedia.org/wiki/Complex_event_processing
This isn't necessarily an issue, complex constraint matching is typically a fundamental part of the ingest path anyway given the algorithms used; making it support more generalized CEP is a fairly straightforward extension of the same computer science mechanics that make polygon search scale efficiently.
In virtually all real systems, data is aged off after some number of months, either truncated or moved to cold storage. Most applications are about analyzing recent history. Everyone says they want to store the data online forever but then they calculate how much it will cost to keep exabytes of data online and financial reality sets in. Several tens of petabytes is a more typical data model given current platform capabilities. Expensive but manageable.
The more fundamental point that the GP is making is that the realm of industrial sensor data scales in ways that people haven't really grasped yet. It's much less about brute storage than it is about the interplay between bandwidth, storage, and concurrent processing power.
So, the problem is that you threw away 90% of your data, and that's where the problem was. Oops. Now you have to switch on "Save all the data" and hope it repeats. So, given that you have to have a "Save all the data" switch anyhow, you might as well turn it on from the start.
In addition, changepoint analysis is an entire field of research in and unto itself.
Look at how many articles there are about analyzing "Did something break in my web service or am I really doing 10% more real traffic?"
It’s typically a mix of everything, but predictive maintenance, anomaly detection and failure analysis are the most common. For example, there is one process that does trend analysis and tries to “predict” acceptable boundaries of a certain sensor’s measurements, and this is then compared in real-time with the actual sensor readings. If things fail for some reason, a technical engineer will dive into the data with dashboards (think: Grafana), zoom in, compare the readings with other sensors, etc.
The sheer volume of the data makes it fairly painful. Downsampling does happen, but only after a few weeks. This means that you still need enough storage capacity to deal with the full stream of data in real-time.
Most of the analysis that is done usually falls under one of two categories. First, inferring (you can rarely measure it directly) when something has changed in the real world that is relevant to your business so that you can adapt to it immediately -- the applies to everything from autonomous driving to agricultural supply chains. Second, detecting anomalies -- the unknown unknowns -- so that risks can be managed when the real world appears to not conform to the models upon which you base decisions. A third category is support of industrial automation, which benefits immensely from high-resolution multimodal sensor data models, though this is largely a cost reduction measure. These categories are hand-wavy but in practice, boring industrial companies have concrete metrics they are trying to achieve or risks they are trying to manage in the most efficient way possible.
Having "cost" on the board as a factor we were actively trying to optimize for during design pulled us in a direction that is quite foreign compared to off-the shelf solutions.
That last paragraph rings true -- one of our big challenges specifically was in ingesting and indexing data that needs to be queried across multiple dimensions, things like aircraft or drone position telemetry. But once we found a workable solution for that, it specializes quite well to simpler workloads very well.
I think this is due to the nature of the mentioned companies are not being too common (yet?). There are tools and systems that you can use, especially from high frequency trading which has somewhat similar challenges. KDB+ and co. would be my first stop to check if there is something that I could use. The question is the financial structure and scaling of the problem, to determine if these tools are in game. There are other interesting projects in the space:
- https://github.com/real-logic/aeron
- https://lmax-exchange.github.io/disruptor/
Of course these are not exactly what you need, long term storage and querying (like KDB) is largely unsolved.
The other tools that you might be referring to by "most of the opensource platforms" indeed are not capable doing this. I spent the last 10 years on optimizing such platforms but it is not even remotely close to what you need, you (or anybody who thinks these could be optimized) are wasting your time.
We do this. Have a load balancer with a fleet of nginx machines insert into bigquery. Inserts scale well and the large queries work since it is columnar. The issue is price. It's terribly expensive.
I've used the following pattern in the past: - generally only send batched data in as large an interval as possible - if somebody looks at a device, immediately (well, might take some seconds) query the batched data and switch device to a "live" mode that provides live data instead of "wait and batch".
This will be a bad idea for scenarios where there's a reasonable expectation of surges of people needing "live" access, but for our use cases of industrial data, it works very well. We only watch our own devices, which are in the lower tens of thousands, but I don't see why this should not scale to more, under the restrictions mentioned above.
As a systems engineer with a good track record and an interest in starting an endeavor, this is a very attractive statement to me.
Where can I read more about how the sensor networks are configured, the use-cases, etc? I'd like to read into this a bit more.
Every use case has unique data model requirements (minimal standardization, different sectors) but there are easily identifiable platform components that almost everyone needs which aren't available. Surprisingly "simple" architectural holes would be a scalable business if competently plugged, the perfect MVP. These enterprises have an aversion to developing software, it isn’t their strength, and they know precisely how many millions per year a real platform would save them -- value is concrete. However, they are also technically sophisticated as to why all existing platforms fail for them, you can’t fake understanding the problem. I have the benefit of having worked on this market problem for several dozen organizations over the last 15 years, ranging from Big Tech to small EU industrials, so I see it more from their side.
Little is written about it. Everyone is essentially trying to use diverse multimodal sensor data sources to paint an accurate model of some part of the physical world in as close to real-time as possible. Easy to say, very challenging to do. Sometimes these data models are not about their business per se, their hardware puts them in an excellent position to build them so that they can sell it as a service to businesses that can actually use it. Often overlooked is that there are extremely difficult computer science problems with little public literature buried in the design of such systems, and expertise in this computer science is critical to being successful at it. Virtually all startups that try to enter this market completely botch the technical execution, assuming that these platforms don’t exist as a function of business execution when it is actually a hardcore tech startup. The technical execution expertise is the real moat for this business, everyone underestimates how deep that rabbit hole goes.
FWIW, I’ve been laying the groundwork to build a startup in this space for a while now, I even purchased a very good .com domain. :) Bespoke implementations at several highly recognizable organizations are based on licensed code components I designed. There is a massive demand overhang and the market was ready yesterday. The broader ecosystem has room for several startups to coexist, there are many niches currently unfilled.
What piece of architecture you think is left out?
Let’s step back for a second and just acknowledge that you’re in a very narrow slice of the market. The number of companies that are paying $100M/year to store sensor data is probably countable with 8 bits.
So it might seem like a large gap for you, but it’s honestly not relevant for 99.99% or developers.
It isn't a solved problem but people are working on it.
I guess you could have a time series database that used compression but I don't know of databases that do
If you add real-time sensor data sources to the mix, the rest of the architecture model kind of falls apart. Requirements upstream have cascading effects on architecture downstream. The deficiencies are both technical and economic.
First, you need a single ordinary server (like EC2) to be able to ingest, transform, and store about 10M events per second continuously, while making that data fully online for basic queries. You can’t afford the latency overhead and systems cost of these being separate systems. You need this efficiency because the raw source may be 1B events per second; even at that rate, you’ll need a fantastic cluster architecture. Most of the open source platforms tap out at 100k events per second per server for these kinds of mixed workloads and no one can afford to run 20k+ servers because the software architecture is throughput limited (never mind the cluster management aspects at that scale).
Second, storage cost and data motion are the primary culprits that make these data models uneconomical. Open source tends to be profligate in these dimensions, and when you routinely operate on endless petabytes of data, it makes the entire enterprise problematic. To be fair, this is not to blame open source platforms per se, they were never designed for workloads where storage and latency costs were critical for viability. It can be done, but it was never a priority and you would design the software very differently if it was.
I will make a prediction. When software that can address sensor data models becomes a platform instead of bespoke, it will eat the lunch of a lot of adjacent data platforms that aren’t targeted at sensor data for a simple reason: the extreme operational efficiency of data infrastructure required to handle sensor data models applies just as much to any other data model, there simply hasn’t been an existential economic incentive to build it for those other data models. I've seen this happen several times; someone pays for bespoke sensor data infrastructure and realizes they can adapt it to run their large-scale web analytics (or whatever) many times faster and at a fraction of the infrastructure cost, even though it wasn't designed for it. And it works.