dave_ops
Having now screened over 300 candidates for openings on my team, the overwhelming majority of them being people already in the Bay Area, about the only thing I can say is that with all the easy money sloshing around the economy there it's just way, waaaaaaay to easy to be a terrible engineer and still maintain being a highly paid "professional". Most of the people I screen/interview are advanced-IDE users at best or simply framework operators (i.e. they have no idea how the things even work that they're claiming to be experts in).
Sure there are a lot of bright folks too (law of averages and all that), but it's depressing to see what people are getting paid to do glorified web scripting or whose entire job function could be replaced with 50-lines of Bash.
foo(bar) -> void;
foo(X) -> undefined.
foo(X) -> foo1(X).That doesn't depend on the "technical" part of "technical leadership".
Most of the cases where I run into this problem and end up feeling like I didn't gain much, if anything, from doing a straight C implementation are situations with deeply nested data structures.
I get that the thinking is that the benefit is that once I've made the compiler stop complaining, my memory management model should at least be sound and safe, and that is a win. Though immediately following that I start to have dream-like fantasies where this entire static analysis stage is simply bolted onto C instead of being a whole new language.
Afterall you can do some pretty impressive stuff with nothing but a pile of preprocessor macros (see Objective C).
Lifetime elision seems thoroughly broken or extremely limited in functionality, and now when I'm writing (sometimes seemingly trivial) solutions in Rust I'm spending at least as much time and mental energy explicitly annotating lifetimes as I would be if I was just managing malloc and free in C.
As a result the pain vs. benefit curve doesn't bend nearly as far toward Rust as it theoretically should.
They in fact could have solved this problem with Rails or even a more full-fledged CMS like BrowserCMS.
An on-prem corporate twitter would actually be useful.
Stick about 10GB of small entries in it (should be enough to create all the levels) and then see what happens.
Also, you could reserve the persisted [H|T] for controlled shutdown scenarios. Basically anything that isn't complete system failure if you're properly trapping signals.
But moving the concern to the consumer to track the cursor doesn't make the protocol any more stable. To keep a stable cursor, the consumer would need to persist that someplace, which just pushes the acknowledgement to that persistence component instead. If a stable cursor is what you're after, then co-locating it with the durable queue provides a simpler solution with a slightly better consistency guarantee.
The garbage collection problem is a real one, but realistically how many consumers is an infrastructure service like this going to have? Tens? Hundreds? Thousands? Millions? Billions?
No matter which one of those you pick it's a trivially small secondary index to maintain even if you never reaped it. I mean it's a K/V problem (consumer_id -> queue_offest) and there's a K/V store already sitting there. If you didn't want it to grow forever then you could establish a TTL policy via configuration.
The problem you would have is consumers that don't have stable or bounded id's. Like a system that assigns a new id every time the consumer makes a request or the consumer is restarted.
But deletes are going to have a big impact still, and (working from my failing memory of LevelDB internals) I think might actually be the pathologically sad case.
So the queue keeps track of the high-watermark on a per consumer basis and all the consumer has to do is show up, tell the queue its deterministic name/id (might be driven by imaging, configuration, or SDN), and the queue will serve up the next new item that consumer hasn't seen yet.
This would be handy for really dynamic transient worker topologies because it keeps the mutable state and state tracking concerns entirely outside the transient worker.
That said, I still wouldn't use LevelDB. Unless I was expecting to do multi-attribute range queries or something (now we're well outside queue territory), but even then you're still folding over the data for knowable start/end markers and a linear scan over a binary term file will be faster than the multiple seeks + segment scans that LevelDB requires.
Since the whole thing was designed to be an OS unto itself, it has the concept of "Applications" and "Releases" that come from OTP.
It's trivially easy to make things properly decoupled by dividing logical functionality into "Applications" and making a game-time call about whether or not they should be bundled in the release and consumed sorta like a library or run on a completely different VM/node and consumed like a service. You're architecturally incentivized to make "microservices", but fully able to couple or decouple them at will while maintaining the same semantics in how you consume them.
I mean it's log/event aggregation for ops insight. Unless the whole system is some tightly coupled feedback loop into an unsupervised machine learning model where the whole thing has actual hard-real-time requirements (something which might well be impossible to build), then there's no possible way that having a second or two delay, between a message being created and when you can actually see it, can possibly matter.
I mean you don't have someone with instantaneous reflexes and resolution ability sitting there 24/7 with their eyes peeled as a stream of thousands of log messages flies by.
The whole premise seems spurious. Is it really necessary for every startup and their uncle to delude themselves into thinking that their use case is "mission critical" or "carrier grade" or whatever?
Auth0 provides basically "login as a service". Its not like they're managing the access control to nuclear launch codes or something.
Unless some medical device manufacturer was stupid enough to make a critical surgical assistance device require an internet connection, a WAN round-trip on unreliable networks, and reliance on a 3rd party service in order to start operating it... how can this service being down possibly be anything more than an annoyance? What's the worst possible scenario? A session has to be rebuilt? A user has to make an extra login attempt?
By their own admission the service has gone down already due to the old system architecture. How many babies died?
We become fast friends and feel like we really understand each other.
But days pass, and each encounter feels less magical. It's almost like we having nothing in common. Like we're from two completely different worlds. One where its stuck in the past and one where I'm ambitious and excited about the future.
After awhile we don't really speak to each other anymore, and after some pretty ugly fights at work that get too personal... I rewrite it.
There are some things you have to do right from the beginning or you're never going to get them right.
Containerization isn't new. It just has a brand name now, and the "all the way" solution to this problem is unikernels.
So I finally just started using Emacs instead. Setup my keybindings so they're Mac/Sublime Text like and wrote a little custom frame/window manager in elisp to make the application behave the way I want, and I'm free from having to worry about one of my core tools turning into abandonware.