xyz-x

Does anyone have experience using Apache Iceberg, Hudi or DeltaLake for storing logs instead of Loki?

Monads but more importantly MonadTransformers so you can program in a legible fashion.

However, there's a lot of manual labour to stuff everything into a monad, and then extract it and pattern match when your libraries don't match your choice of control flow monad(s)!

This is where I'd prefer if compilers could come in.

Imagine being in the bowels of a DB lib, and realising that the function you just write might be well positioned to terminate the TCP connection that it's using to talk to the database with. Oh no: now you have to update the signature and every single call-site for its parent, and its parent, and...

Instead, it would be neat if the compiler could treat things you deem cross-cutting as a graph traversal problem instead; call a cancelable method and all callers are automatically cancelable. Decisions about whether to spawn a cancelable subtree, to 'protect' some execution or set a deadline is then written on an opt-in basis per function; all functions compose. The compiler can visualise the tree of cancellation (or hierachical loggers, or OT spans, or actors, or green fibers, or ...) and it can enforce the global invariant that the entry-point captures SIGINT (or sets up logging, or sets up a tracer, or ...).

So imagine the infrastructure of a monad transformer, but available per-function on an opt-in basis. If you write your function to have a cleanup on cancellation, or write logs around any asynchronous barrier, the fiddly details of stuffing the monad is done by the compiler and optionally visualised and explained in the IDE. Your code doesn't have to opt-in, so you can make each function very clean.

Too bad Apple hasn't released Apple AI at all; why bother with press releases featuring it? (EU)

Seems to be it's an ebb and flow of complexity; sometimes it increases like with sidecar proxies and their issues, like how they might listen on NICs differently, or how they cause grey failures, and sometimes it decreases like when the sidecar moves to the kernel and gets rewritten in eBPF and we can remove another ip filtering-firewall-chain. So right now it seems to be a bit of an ebb, as these things are becoming moduralised.

We ran Zalando Operator for Postgres in k8s for a year, until finally succumbing to its technical debt that leaks out from every bit of its software being.

After switching to the Chrunchy Data pg operator v5 on k8s, we've had close to zero problems - one or two times a year the log shipping / HA replication fails and we have to restart it, but it's really neat! I can *warmly* recommend it; it really is CloudSQL in K8S.

Are these side-channel based management technologies turns on even on MacBook laptops?

It doesn't have to have a turing complete interpreter; it just have to be provably terminating, and you can build most use-cases as an active message.

What I'm after with the monadic control flow are the error-cases; let's say you have

> music.getPlaylist(ps => ps.userId == "u123").findTopSongs(10).enqueue(qInstance) => Result<C, Error>

it would be nice to see how this would be interpreted into an AST and executed as an active message on the server (receiver).

That said, I brought it up because the copy alluded to it. It's a great time sink to build an interpreter, even if it's only acting on a unit of work whose variant is strictly decreasing, just look at Linq-to-SQL and IQbservable<T> a decade ago.

> KJ Promises

Side note: another copy that greatly frustrates me, as I now try to find the docs on the above async stuff:

> Essentially, in our quest to avoid latency, we’ve resorted to using a singleton-ish design, and singletons are evil [linked to a page that crashes for HTTPS-everywhere users (me) with PR_END_OF_FILE_ERROR].

(Besides the broken link,) singletons are not always evil. I know you know this, I know the copy is tounge-in-cheek again and being ironic — BUT POE's LAW FOR CRYING OUT LOUD :D https://en.wikipedia.org/wiki/Poe%27s_law — "Poe's law is an adage of Internet culture stating that, without a clear indicator of the author's intent, it is impossible to create a parody of extreme views so obviously exaggerated that it cannot be mistaken by some readers for a sincere expression of the views being parodied"

...so KJ Promises; I can't find that mentioned. I only find Promise Pipelining; but that must be your Op:s that allow for field traversal? The site has this copy:

> [RPC Page] With pipelining, our 4-step example can be automatically reduced to a single round trip with no need to change our interface at all.

But I must be from another planet, because I really don't understand:

- first you have a pretty decent design of files that mimic local files

- now you instead showcase what rich messages look like, calling that a "[message?] singleton", linking to a broken site

- path string manipulation exists in every standard lib, it's not something we implement

- if someone wants to perform multiple ops on a file - let's say read a chunk of it * only Data needs to be resused for there to be no copies (contrary to the copy), but that's also a problem in the first decoupled example

  * often, almost always, when I read about what an RPC system can do, I'm not in the memory-management mind-set, so I don't care about re-allocating resources

  * caching is not a relevant solution, it's to the contrary; completely irrelevant in this context and it's detracting from understanding what you want me to understand

  * caches aren't error-prone when used right, like with immutable data, or read-through caches as transparent proxies can do, but all of this is beside the point

- then there's a discussion about "giving a Filesystem" to someone, when it's really all in my program

  * hard-coding a path is out of scope; that's about engineering process, not about the software. You ask "But what if they [have] hard-coded some path", I answer "yes, so what?"

  * what if we don't trust them (our own code?) — no, it's not an AuthZ decision locally, it's remotely, so you want to be explicit about the error cases here, but there's nothing about it — instead the copy says "now we have to implement [authN authZ systems]" — but again, this has nothing to do with merging small interfaces into a larger interface; it's a problem even with the small interfaces

- the section ends with the broken link and then the next section states "Promise Pipelining solves all of this!" — but noo, there's so much mentioned above, and the premise is unclear and I have no idea what exactly promise pipelining solves!

And then you go with the calculator example; but the file is large and I still don't know what "Promise Pipelining" means, where to look. I see a lot of construction of values going on and then a blocking wait (polling of the event-loop, but that's also not the point). There are so many bugs in that copy, that it's really hard to know where to start detangling it! With that copy, I would never in my life touch the underlying code! It should be fixed! (sorry, I'm getting into a state here, but that copy... wow)

And this is the kicker that seals the deal:

> Didn’t CORBA prove

Ehm, WTF? Why not contrast with gRPC? But also, why not clarify the above first so I can use that understanding myself? If I'm not a newbie at this, why do you mention CORBA? Do I look like the kind of person that would ask that question? It's demeaning to the reader.

And you mention object capabilities; that's a HUGE area of research, of which I've worked with no live system using it. But here it's casually mentioned, like building such as system is a walk in the park without explaining how.

---

So here I am after another frustrated 40 minutes on the site, and I still haven't found the docs on KJ Promises.

> Exactly. Look, here's what I think happened here: You looked at this page, you saw the silly marketing, and your "gut feeling" told you that this guy is an amateur who needed to be put in his place. So then you started seeking out details that you could criticize as amateurish.

Again, I'm not attacking you as a person or your work, I'm attacking the shoddy copy. But yes, this whole thread is about what the first impression of your web site is like, and it's amateurish. Perhaps you're someone who can live with that, and how that results in lost conversions (?) from visits, maybe you're not. In any case, from running businesses myself, I know that getting clear and honest feedback is worth its weight in gold.

> instead of actually digging into every detail in depth

Which is precisely my point that I won't do. If you can't communicate clearly to your visitors on their first visit, but expect them to dig into the details, you've lost that battle.

> For example, you assumed that KJ promises don't use monadic control flow, when they most emphatically do. How did you get there? You looked at some code and found one example that didn't happen to use the monadic flow.

Again, what I found has nothing to do with you. I'm yet to actually have seen the examples you mention in your docs, and the pages I've linked to and the calculator sample doesn't show your statements to be true (only based on that). Perhaps, instead of being so tounge-in-cheek on your introduction page, you assume the visitor is not an amateur, and show what you got (you can move the funny bits to after the introduction, when the reader is in on the joke)?

> The "Return" message from each call serves as an application-level acknowledgment that the message has been received and processed. This is enough information for the sender to maintain a send window that places an upper bound on buffer bloat.

THIS; is what kind of copy you could use. Also not using a confusing `Data` record name, which conflates L4 and L7 semantics (even if that data is identical to memory of a record), not using `my-` prefixed variables, but building a use-case like "suppose this was for streaming music", or something.

If uptake is an aim, focusing much more on other languages than C++ is probably wise, too. And some diagrams to explain what trade-offs have been made (can you load-balance CapNProto like you can gRPC? Is -Proto, Protobuf, so I can use an existing schema registry in my "enterprise"? Is it safe, "like HTTP2" — the newbie stuff in relation to the world around you).

> Cool, I just hit the 30-year mark myself.

Time flies, eh? ;)

> But my advice for you is, next time, try to recognize when you don't actually know the details, and then try to assume the best possible details, or at least ask questions, rather than assuming the worst.

I'm aware I don't know the details! I've said so repeatedly; if I read the details while I have a conversation about how someone who's never encountered Cap'n Proto before, reacts to its material, I'm not that person anymore.

From the above person's perspective, you have a bit of a problem with marketing; I've actually bounced off your front page about three times the last six (maybe?) years and the copy/messaging has always put me off. I'm sure I'm not the only one. Just ping me if you actually do like having someone review that copy for you.

> it turns out they haven't actually performed the compile-time indirection

Here's how much monadic control flow he actually has in Cap'n Proto: https://github.com/capnproto/capnproto/blob/77f20b4652e51b5a... — nothing except fields. Gut feeling was correct then.

https://news.ycombinator.com/item?id=22972728

I do really appreciate that you try to explain this from first principles. First, to explain myself, I'm primarily programming in F#, which is a hybrid functional AND object oritented lang (also use: TS, Elm, Haskell, C#, Python, Ruby etc). I prefer to write the software in a functional style though, so I rather use closures and self-recusive functions acting as message loops over the dynamic dispatch of "telling" objects to do things.

> and, yes, passing around functions as values

If we go back to the foundations of what I would call functional programming, Lambda Calculus as defined by Church, I like to give the evaluation of booleans as a prime example of how functional programming computes by passing callbacks around: https://en.wikipedia.org/wiki/Lambda_calculus#Logic_and_pred...

Since this was arguably the very first functional programming "language" that existed in the world (even: one of the very ways to do structured computation!), and is almost exclusively uses callbacks, I'm very confident in saying that callbacks is more functional programming than anything else.

Functional programming still computes heavily via function composition; the principle of passing callbacks around is at the very center of functional programming. To name a few instances of callbacks being passed around; via currying "context" values, by passing a function to "act in a context" (hole-in-the-middle/delegator) or closing (closures) over mutable state to enable runtime re-configuration.

What most people don't realise about functional programming (until they've done it a long time) is that it's not in the syntactic features, nor in the type system, that functional programming really shines, it's in the compositionality of it.

> But a particularly surprising one is that often the order of statements don't make any difference

This is generally a false statement. You can mean many things here: dataflow programming (https://en.wikipedia.org/wiki/Dataflow_programming) or the ability of monadic computation expression to choose when in time it executes its built-up construction, or how lazy evaluation can share chunks (bits of compute) that are beta-(normalised)-normal to each other.

What is true, however, is that callbacks is not ONLY a core concept in functional programming (even if it's arguably MORE core in FP); even PL/1 and Cobol have address pointers that gets passed as callbacks, but there it's more used for cross-system interop than for composition. Assembly and even C also heavily use callbacks, but those are not OOP. The dynamic dispatch you have in object hierarchies in OOP, is not even such a close concept to callbacks (https://en.wikipedia.org/wiki/Dynamic_dispatch).

> write a series of instructions that are executed top-to-bottom

That's procedural programming, not OOP: https://en.wikipedia.org/wiki/Procedural_programming

---

> I think the parent commenter called you out on this even though it's not the main thrust of your argument because it's a really significant misuse of terminology and shows a lack of understanding of fundamental programming concepts. Pedantically it's a bit of an ad hominem attack, but you seem to be coming down hard on Cap'n Proto especially because of its misuse of terminology and it's ironic that you're the one misusing it.

Obviously I disagree with this (see above, with references!). The way he answered is unconstructive and only leads to hard feelings. I'm coming down hard on the way the Cap'n Proto article is written, and I can back it up and argue my point. I'm in no way attacking the author, or objectively saying that Cap'n Proto is bad — only that I think the way it's making its case is.

> Another example of this is where you object to "streaming"

I'm objecting not to "streaming" but to how the code makes it look. It's really unclear to someone who doesn't know the Cap'n Proto syntax. I think you're missing the point that my review is from someone who really understands these concepts, reading the article and forming an opinion of whether to give Cap'n Proto a chance based on it. Also, the article only says streaming means "returning multiple responses", which is an API-level concern, not a protocol concern (* * not a protocol concern _ necessarily _ * * : the protocol is what should be explained; for example gRPC uses HTTP2 push to do streaming responses)

> You seem to be talking as if this page is a first introduction to Cap'n Proto, when it's not, it's just a changelog entry intended for people that already know what the library is

It is the first introduction I had to Cap'n Proto, and I also after reading the article read the main site's introduction. I'm giving comments on how I perceive Cap'n Proto based on that.

Very nice of people to down-vote because they don't agree with what I write; I'm not being mean or evil here, I'm just stating an opinion and trying to back that opinion up with clear reasoning, references and links to actual research. Down-voting then seems very much like group-think to me.

> It's not polling. The idea is that the callback is called multiple times to send all the chunks, for one invocation of `streamingCall()`. Sorry if that wasn't clear.

The point of my comment was exactly that there were many things in the release notes (and subsequent visits to the site) which were unclear.

> As others have pointed out, you are taking a very superficial and literalist definition of OOP and FP.

Precisely, which is my point; why use that terminology at all? It's neither specifically OOP nor FP, but if it's something, it's more FP than OOP to program with functions. Either way, it's super-superficial (pun intended! :)) and detracts from your message in your release notes!

> `Data` is a basic data type in Cap'n Proto. It means an array of bytes. You can use any other type here if you want, it's just an example.

But my point is whether the RPC frameworks garantuees complete messages or only complete "chunks" to arrive atomically?

> What is "not the case"? The whole point of this streaming feature is to do exactly what's described in your Wikipedia quote.

You're not solving the problem as far as I can read in your article, you've only pointed it out. In my second link, you'll see there are both a) second-, third- etc hop buffers to take into consideration at the sending side, b) there's timing to take into consideration, c) there's the whole "send a large message initially" heuristic, to take into consideration; all in all it turns into the end-to-end argument — you need knowledge on both sides to optimise this, and preferrably an out of band channel.

> "Time travel" and "infinitely faster" are obviously tongue-in-cheek claims.

It's obvious the literal claims are, but it may not be obvious to readers that your trace diagram is also tounge-in-cheek; at least clearly break with the jokey-attitude afterwards and perhaps show an actual trace diagram of how it works (because the trace diagram is only partially a lie; the pipelineing is still an interesting concept that I think people want to understand!).

> No, the calling thread does not block.

In your sample here you block https://github.com/capnproto/capnproto/blob/master/c++/sampl... to get the result. What I'm saying is that it's very unclear how you compose your flow without doing the blocking wait to get intemediate results in between RPC calls. (Exactly: you seem to lack a monadic control flow, specifically the tie-in that endofunctors would give you, and given this lack, you cannot handle exceptions out of band to the happy-path via Choice/Either/Result types)

> Haha yeah that's me, just some amateur that knows nothing about network protocols...

No, it's not nothing about you. I'm sure you're very competent. I'm sure Can'n'Proto is competent too; what I'm talking about is the way the framework is described and how the messaging looks to someone with close to two decades of programming experience. I hope you take is like this; as constructure criticism. I'm already very happy that you replied, because just as I believe you're a stanch proponent of your RPC framework and believe in it, I believe in my gut feeling when it comes to software — and my gut is telling me the above about your marketing message.

Passing around pointers to functions is more functional programming than object oriented programming. You're literally programming by passing functions around with callbacks. But obviously, this is not the main point of the comment; the argument that it's "like OOP" (which it's not), is what I'm attacking.

For me, new to cap'n'proto, this blog post doesn't cut the mustard because of multiple red flags:

- to start, it's self-congratulatory at stating that streaming already exists "[via] promise pipelining" — but that has a name, it's called polling, not streaming. Making asynchronicity explicit doesn't make a protocol streaming.

- in the same paragraph an "object-capability model" is introduced as a concept, but not explained

- second paragraph: "think of this like providing a callback function in an object-oriented language", when it should be "in a functional programming language" (callbacks aren't OOP, they are per definition functional programming)

- second paragraph, vocab: what's a "temporary RPC object"? Contrary to the precise, albeit unexplained vocabulary, in the first pragraph, this is vague.

- creating examples with "MyInterface" being the service shows a lack of creativity and a rather low ability to communicate; is this service on the server side, or the client side? Noone knows, and "Callback" is not a good name for a callback, it should be "SendEmailWithData" or something that makes sense.

- `sendChunk @0 (chunk :Data)` doesn't make sense to a beginner without explaination; what's `@0` and why do I care?

Here's what made me write this comment despite the threshold annoyance in commenting:

- Why name a message, `Data` when it's clearly NOT a chunk of data in layer-3/layer-4, but rather a layer-7 artifact with retries and checksumming implemented to ensure complete and accurate message delivery?

Finally, the articles goes on and discusses control flow via a proxy variable; your OS'es TCP send buffer size. But the linked Wikipedia article states:

> because the protocol can only achieve optimum throughput if a sender sends a sufficiently large quantity of data before being required to stop and wait until a confirming message is received from the receiver, acknowledging successful receipt of that data

Which is not the case for Cap'n'Proto (admittedly it states it uses a hack). And there's no discussion of end-to-end problems like BufferBloat, which are very hard to solve by only looking at your own buffer https://en.wikipedia.org/wiki/Bufferbloat#Solutions_and_miti... — or even the semantics of "blocking on server's return value" (Is it enough for the receiving process to have the message in memory? The type system showcased seems to tell that story)

The article also doesn't state how a simple RPC call works. Going to https://capnproto.org/rpc.html immediately puts me off by inventing "time travel", calling it "promise pipelining" and showing an impossible trace diagram (you can't have messages go backwards in time).

But when explaining it, it's really RPC message coalescing and compile-time reference indirection, from the promise to the underlying object instance as it is after executing the coalesced message pipeline. However, even when using the example of a file system (which is about as exception-intense as you can imagine), exceptions are ignored.

Looking through the Calculator example (https://github.com/capnproto/capnproto/blob/master/c++/sampl...) it turns out they haven't actually performed the compile-time indirection, but actually block the calling thread like any random do-it-yourself-RPC framework out there.

What a RPC framework should do is give you:

- an extremely clear serialisation model that is outside of the framework

- a clear API

- clear garantuees / invariants on how it manages the complexities of network programming

In short: it must be very clear in what it promises. Cap'n'proto is not.

Hasura on Postgres works really well too.

Does anybody know why the USA hogs the toplevel domain? It's not the only government in the world. It would seem more just to make it more like .com than .edu.

Like their free DNS service that supports DNSSEC, Rotor (CDN JS optimisation), Let's Encrypt integration out of the box, low latencies where the users live, etc

About 7 years ago, when I was trying to monitor .Net apps, there weren't that many alternatives available. The Elasticsearch-Logstash-Kibana stack had just gotten off the ground and I needed a way to send it structured logs and a large number of machines. Logary.tech was born to solve that problem.

Since then Logary has expanded with excellent support for sending both metrics and tracing data to a large number of targets. In production, I use this setup;

client browser -> Logs & metrics to Logary Rutta HTTP ingestion endpoint via Logary JS

nginx-ingress -> Traces to Jaeger Agent via opentracing C++ client nginx-ingress -> Metrics to InfluxDB nginx-ingress <- Metrics via Prometheus scrape annotation

Our NextJS site and GraphQL server: site -> Traces to Jaeger Agent via opentracing site -> stdout logs via Logary JS (also get added as Logs in the Span of OpenTracing) site <- Metrics via Prometheus scrape annotation and prom-client

api -> Traces to Jaeger Agent via Logary's F# API api -> Metrics to InfluxDB via Logary's F# API api <- Metrics via Prometheus scrape annotation and Logary.Prometheus api -> Events to Mixpanel via Logary.Targets.Mixpanel api -> Logs to Stackdriver via Logary.Targets.Stackdriver (hosted on GCP)

Also, Kubernetes ships logs via FluentD to Stackdriver in GCP, but they are not structured, and the remaining infrastructural services also send traces to Jaeger if they can.

Logary Rutta is a stand-alone log router, written in Hopac + F# (like Concurrent ML), and used by some of the largest Swedish software companies for thousands of logs and metrics per second. It's capable of shipping to a large number of targets https://github.com/logary/logary/tree/master/src/targets Since it talks HTTP and UDP with a number of encodings (JSON, plain, binary), it's easy to plug into an existing infrastructure and existing log shippers. It can also connect point-to-point to itself with a high-perf binary encoding. Because you can send any JSON into it, it's very easy to get started with together with mobile apps.

Logary for JS currently has support for user logs, and I'm currently testing rudimentary metrics and browser info.

Logary for .Net supports the OpenTelemetry spec, structured logging and metrics.

Of course you can pick any toolchain you want, but I've had great success (and great fun!) writing and using the above. You can see I don't keep logs on disk; it causes them to fill up; if your network is down, your service is down, and then you know it's the network anyway.

Once in Logary, you can choose where you send them. I've done an analytics/ETL pipeline based on Logary with its Stackdriver+BigQuery+GooglePubSub targets and with Flink, with great success as well. Logary is free to use for non-profit and then I have a pricing calculator on the home page, for when you start selling the software you build. Pricing aside, how Logary is structured and how I've used it might give you some hints on how to do it yourself.

> But if you had the key cached, and it changed, you’d probably freak out.

Not in the servers-as-cattle age. By default, a rebuilt server will have a new key. Otherwise, you'd have to save the server SSH key in your configuration/build files, and then you've moved what you have to protect to the source control of the servers, and probably exposed that secret key to many more people and developers than you would have done by leaving the key on the server.

Jumping one stratum forward, with hosted k8s you don't even know the host's key; you do everything via HTTPS and the almost globally accepted list of secure CA:s.

It's missing a clear description what those "unique features" are, both from using Racket and to a visitor considering using it. In short, the site doesn't sell properly.

keepalived is never the answer; if you can run it, your services are by definition crash-only share-nothing or inconsistent by design, or else you wouldn't let keepalived choose when to move the "primary flag" to the other service (as there'd be no way of sending the last ACKed data from the previous primary). Since this is the case, you could just load balance across the services and have them both active.

From a networking perspective, getting VRRP working on anything but physical equipment (e.g. in the cloud) is a fool's errand; it's L7/API-based and not on the ethernet level. Similarly with keepalived, which will get isolated from the monitored instances (thereby failing to the other, also "down" instance) — except it might have access to the API gateway of the cloud provider thereby disassociating the V-IP from both your instances; so you'll end up with more downtime with keepalived than you gain by it.

Since DNS is by default inconsistent, but eventually consistent and thereby possible to load-balance, you could run one instance of this stack on your static home IP and another instance on GCP/DO/AWS and configure multiple DNS servers in your DHCP options and on your phone, to get higher availability.

This was really well made and fun to play! Thanks for making it!

Makes me want to create a game for CSP where the goal is to "make it right" (some concurrent condition) rather than "make it crash" (buggy usage of primitives).

Sometimes even a single person at a keyboard writes tests, even without large development teams.

Let's try another comparison: that to a skilled surgeon. 4+4 years to get GP status, then at least 4 years more to become surgeon; another 2-10 to lead in an operating theatre.

Diseases vary just like reasons for misbehaviour in code. While coding also has a bit of construction, that surgery doesn't, a lot of coding time is spent on extending or fixing existing behaviour. As such, the simile is better than the bridge-construction one.

It would also allow discussing leadership; a strong lead handles the 1-12 hours the surgery takes. He/She's expected to know the human body as a system, by being able to diagnose adverse conditions that occur during the operation, instructing the people around her as she goes.

Operations can't take too long, or fatigue gets you; the corollary being that you don't have slippage like you do in s/w dev. You can't be too unskilled, or else you can't perform the work. You have to have lots of training before you're the one leading.

Contrasting education; a lot of physicians go through much rote memorisation when they start their education. Then they continue with laborations; this could be useful for software engineers and operations folk, by letting them try their hand at diagnosing production systems having problems. Such training could be done by a 20-questions approach, with each question being answered a metric of a category of log entries. In the end you should know what the problem was. Labs at uni are much more constrained; they don't teach how large production systems behave and don't teach mental tools to debug them, they only teach the basics of programming. Furthermore, what they teach of programming is never geared towards what real systems look like, because most teachers have never been close to one.

There's no-place in the world that you can be education like what a surgeon would get; comp-sci is more like training everyone to be a psychologist (because we want to get the full picture!). Apprenticeship programs are more like training to be a nurse.

- The formal education people receive is not applied towards bettering industry performance, it's geared towards inventing and academia - Shorter education people receive is not about understanding the system and how to construct and fix it; it's about working next to it - Similar hard rote-memorisation tests could be coupled with in depth debugging/operations sessions and experienced teachers active in large production systems, like understudies preparing to be expert surgeons - In the light of this; we need a career ladder that doesn't end with the same title you start with "engineer"/"senior engineer". One that is structured.

Broken article

I used RMQ because I just finished a smaller study of its properties when running it on Kube.

In fact, its clustering on Kubernetes is really well done. It's very easy to get up and running and to use from an operations standpoint.

`pause_minority` yes, so there's a setting for this. No safe HA mode? Could you explain that further? It is not entailed by your previous statements.

I did a load test in the above mentioned study and each node in the 5-node cluster hovered around 300 mCPU and 250 MiB with a throughput of 10 MiB/s of messages in publisher-confirms + consumer auto-ack + 100 inflight —mode. That was 7x our needs so I left it there for now.

A model in computing does not outlive its usefulness because you say so:

- no replay: not needed; we're contrasting with RPC here which also does not have replay - zero visibility: this is false as there are lots of metrics and libraries focused on AMQP - should be a database: no, it should be a networked queue with atomic broadcast in the happy case (unhappy case: see FLP result) - NACK does not do what you think it does; it puts the message as close to the head of the queue as possible and even putting at the back of the queue like RMQ did around v1 is a valid resolution, because you don't get ordering guarantees nor exactly-once in a distributed queue, generally (Kafka does not give you exactly once [see atomic producers RFC on their Wiki and the consumers are not transactional so they don't consume exactly-once either])

So, I guess someone was wrong on the internet. ;)

Notes:

- You have to use publisher confirms - You have to use a HA-mode of at least exactly=N/2+1 nodes on every queue - You have to use acking consumers - You will get duplicate messages - You will have to implement retries

For simplicity, let's take RMQ

> Multiplying the load on message brokers, often 2-5x, requiring extensive and costly scale out of the messaging system

Scaling it out is no problem. kubectl apply -f statefulset.yaml replicas=5 — done.

Also, message brokers almost never come under load because they are primarily IO bound, not CPU bound. If your broker comes under heavy CPU usage, you're almost certainly doing something wrong.

> No easy flow-control, fast services can produce and consume messages quickly enough to cause side effects for the slow services that are the bottlenecks and dictate overall system performance. This can cause large swings in performance as queues fill up.

In a RPC based architecture flow control would be even worse, cascade up to the caller and cause timeouts and unbounded ephemeral queues that end up looking like a hung system. Worse even, the thundering herd problem will make you unable to get back up.

> Increased latency for end-to-end processing because of repeated round trips to the messaging broker, which must persist messages to disk each time

No, RMQ does not necessarily persist the message each time; you can get a publisher confirm back if all receivers have claimed the message.

Furthermore, while send-then-ACK latency can be up to the disk latency, you're by no means limited to sending one message at a time.

> Increased difficulty isolating performance issues to an individual service vs the messaging brokers vs side-effects from other services impacting the messaging brokers

You almost never have performance issues affecting the brokers.

Suppose you do; then it's most often because you've let your queues grow too long. Where would you be in a HTTP world? You'd be having an outage.

---

> HTTP-based services (REST, gRPC, etc) offer low latency and high throughput compared to queues. They also have strictly lower resource requirements (network, disk, compute, memory) than a message queue solution.

No, not strictly lower; I can imagine a scenario where you have >80% utilisation on your HTTP server on averge; now you'll have growing ephemeral TCP request queues once in a while; then you have to bump the node stats. Consider the alternative; a node that has no latency requirement; it can be utilised to a higher degree; 90-100%. Always at 100%? Then, scale out based on queue size, then scale down again.

For desktop, it would seem the homebrew model is very successful. Products and software do get the updates they need; it's based on a pull-request model and when you create a new package you only have to spend a day to go through the syntax linting process in your pull request, as opposed to a year, waiting for a new stable release in a linux distro.

On the server-side, containers is not primarily about "getting the admin out of the way", but about reducing dependencies of your application to their bare minimums. To its extreme you have Erlang and Mirage that would let you compile your whole application to its own TCP-stack + GC + your app, bypassing the kernel completely.

In that world, there's no need for security patches, because you're using a managed fast language (OCaml/Erlang/Elexir/F*) and you don't have things running next to your application that can pose security problems; no shared libraries, no kernel, no SSH daemon; it's all compiled into your app. You get security updates through your language package manager and they are frequent, because you keep your app up to date as you develop it.

Because your above app runs its own GC, TCP-stack and is self-contained, it now makes sense to move to Kubernetes; because it gives you scheduling, health checks, networking (overlay), a resource permission model (RBAC), a network permission model (Calico) and secret management. Your deployment environment is now your operating system, and the distros aren't needed anymore.

Nice lib and library tests!

The article mentions object orientation several times as a hindrance to unit testing. Perhaps that should be its biggest take-away?

Having written an accounting system, including web/user interface and asynchronous/deferred coordination (such is HTTP/browser programming), for the last three years, I can say that functional programming is increasingly helping my team stay sane.

We do TDD always; sometimes in the form of unit tests; sometimes in the form of integration tests and we tend to write as many of our tests as random/generative tests to avoid having to write large code-bases. I've spent the last two days making a piece of our domain monoidal, having defined the three laws as property/generative tests; the rest of the time is an interactive play with the generator, to see if it can come up with counter-examples, to the code I just wrote.

I normally go about coding by writing a very high-level integration test (at the top-most layer that I still have code in the service/frontend); then I write a huge chunk of the code until I think it's correct and looks pristine and easy to maintain. Now I run the test. If it fails and the test is correct — it tests the right thing and is easy to read — then I start at the top (highest level) of the stack, and write down the assumptions I have thought about while designing the code — as unit tests. Until one passes (at some level n); at the higher level n+1, an assumption/unit test is now broken and I can divide-and-conquer until I find the line of code that doesn't work.

This, together with purification; the methodological extraction of pure functions (things that only have one output ever, for a particular input), makes it possible to avoid testing any side-effects/async things (they simply flow non-async data between pure functions).

This, together with first-level-values for control flow, aka. not using exceptions, and generative/random testing, makes it so that ALL of the input has valid output, makes it so that all functions are total functions. And this in turn makes the code uncrashable and bugfree (for the domains of bugs the above methodology removes).

The domains of bugs that the above doesn't eradicate, are primarily cross-browser bugs on the GUI-side, or UX bugs, where a feature is hard to use/understand. On the server-side we sometimes crash when our logging storage in ElasticSearch goes down and never comes up, the intermediate buffer (Logstash) fills up, and then the app buffer fills up and then the app livelocks, waiting for the logging to drain. (=> operations). The second most frequent reason we have any exceptions/errors/bugs is DNS not working.

The first year of writing the software, we still invoked libraries that threw exceptions, but now we've rewritten them all to do control flow with first-class values, so that is not an issue any longer.

I just wanted to share how we do stuff at qvitoo :), in case it helps anybody.