cangencer

The article is extremely shallow beyond saying "Formal verification a la SPARK" should have been used, while not offering how this could actually work in the real world - I don't think the author has any experience working on any similar piece of software either.

While such techniques are available, would they be really applicable in a very dynamic environment such as with millions of PCs running various windows versions, needing continuous / real-time updates.

And yes, we of course know that QA and testing magically removes all possible failure modes/bugs.

The thread is still wrong, since it was a OOB memory read, not a missing null pointer check as claimed. 0x9c is likely the value that just happened to be in the OOB read.

Yet another contrarian nonsense from DHH - one medium successful product doesn’t give them so much credibility to dismiss what everyone else does.

Like the good old Monty Python cheese shop :) https://youtu.be/Hz1JWzyvv8A

There are actually some patterns to deal with this, such as Saga - I'm actually working on a project (not open-source yet) related to this specific problem. You can reach me at [email protected] if you want to learn more.

Threads are similar to slideshows - you can put a few bullet points that sound reasonably correct, but lacking in detail and context and you mostly forget about it after you read it, it's disposable.

A long-form narrative that is convincing and made to last and read repeatedly is much harder to write.

Your salary is only partially based on "the value created by employee" - most of it is the market forces of supply and demand. When you're remote, you're competing with a much larger number of people for the same positions.

I'll do a shameless plug of our modern Java benchmarking story here for the doubters: https://jet-start.sh/blog/2020/06/09/jdk-gc-benchmarks-part1

The paper is part of the standardization effort but is not the final authority on the process. It is a very good reference on how to approach streaming SQL, even though the Jet model will have a few differences to the presented paper.

p.s.: sorry for the late reply (somehow I wrote and didn't publish).

This is indeed a question that we get asked a lot. We have so far not though about adding more advanced scheduling capabilities for the cooperative threads. With the slot system, if you have 48 core available in the cluster, and running 8 jobs, each job will only use 6 cores each. With cooperative threading, each job runs on all the 48 cores. We have tested something like 5,000 concurrent jobs on same cluster, but essentially they may be competing for the same resources, so you'll need to do your capacity planning accordingly. Simple way to work around that would be to create separate Jet nodes (a Jet node is very lightweight) so you could have separate execution pools.

Do you mean for a single job/pipeline? This wouldn't be possible at the moment. Our current focus has shifted from Beam a little bit - as we found out the beam threading model didn't play nicely at all with Jet's green threads (there is no way to distinguish between blocking and non-blocking calls).

It's not very different than normal SQL. Imagine that instead of a finite result set, you instead have a never ending result set. You can also roughly map operations like windowed aggregation into SQL with some additional syntax. This paper gives a pretty good overview, even though we don't fully agree with the model presented here: https://arxiv.org/abs/1905.12133

This is very true. Stream processing is both old and new and I think it takes time for technology like this to really mature. There's currently a standardisation effort around Streaming SQL which may bear some fruit, but probably still many years away. Right even if you want to use some standard language like SQL to describe streaming queries there's differences in each tool both in syntax and semantics.

While Flink is a fully-featured stream processing framework I think there's some notable differences. Off the top of my mind:

- Flink uses Zookeeper for metadata and coordination, Jet doesn't require any external systems for resilience.

- Flink uses RocksDB and HDFS for checkpointing/snapshotting, Jet stores it in distributed, replicated in-memory store.

- Flink allocates operators to slots, while Jet uses green threads/cooperative multi-threading. This means you can run many concurrent streaming jobs on the same cluster, with very low overhead.

- Jet is basically a single, self-contained JAR. It's all you need to run a production-grade service (+ some connectors, if you'd like)

- Jet can scale up/down with very little friction. You start a couple of processes and they will form a cluster automatically. Kill a couple of the processes, and the cluster goes on.

That said, Flink have a great set of overall features, especially around persistence and huge states. This is another area we're currently investing in as well as SQL support.

The license is meant to prevent service-wrapping by cloud providers, other than that it doesn't have any implications for standard usage. The core library / server is Apache 2 and the rest of the connectors are community license. You can use and embed both the core module and the connectors for free.

The license itself is similar to the licenses from Confluent, Elastic among many others. You can read more about it here: https://hazelcast.org/blog/announcing-the-hazelcast-communit...

+1, in countries like Turkey, Persia, India, Pakistan rice cookers are pretty uncommon. In my home country of Turkey where rice is eaten very commonly, I've never heard of a single person use it. Further, rice is typically first washed several times, fried with a bit of oil and perhaps some spices before being simmered. I don't know if it's possible to do this with a Japanese style rice-cooker.

Follow-up post: https://jet-start.sh/blog/2020/06/23/jdk-gc-benchmarks-remat...

Original article and discussion: https://news.ycombinator.com/item?id=23465660