The irony is that the bricklayers are often paid better, and they can be certain that their profession will remain in demand for the foreseeable future.
The material scientist may hit a dead end, or their company may axe their entire department to "cut costs".
One of the smartest human beings I've ever met ended up at Google, working on something very similar to CRTDs for the Google Wave project. The outcome was a commercial failure and his team was disbanded.
I did the same classes as him at University, and I use almost none of that knowledge in my rather pedestrian job. I make nearly twice what he does and I have nearly perfect job security. So there's that.
Meanwhile, I feel like I'm an adult and can make my own decisions. I don't care if what I like is not "cool" or "hip" or some "up and coming technology". I don't have to be in research or the cutting edge of something to make interesting personal discoveries.
In my career I've accidentally stumbled upon multidimensional OLAP cubes (in the form of SQL Server Analysis Services) and discovered that they're quite fun to work with, actually. The reputation of business analytics as something "dry and boring" is undeserved. It's a wonderful mix of human problems like "manufacturing calendars" and quirky sales bonus calculations on top of fancy optimisation tricks to make calculations over terabytes of data nearly instant.
I've also had the incredibly gratifying experience of making a bog-standard web app with only a few mildly tricky technical bits for a department of education. It helped about a million kids read more than they would have otherwise. That's something. In fact, I'm certain that my one library app made kids read more books they wouldn't have otherwise than a dozen English teachers.
My parents were educators, and I outperformed them a hundred fold without ever stepping foot in a classroom as a teacher.
So yeah there's no shame in working in more mundane & utilitarian areas. And, like you said, you might reach more people with your work that way.
Wave was disbanded, but the technology was apparently integrated into other things such as Google Docs, which uses OT for collaboration quite succesfully (despite the criticisms in OP's article).
> So, having said all of that, is there hope for adding things like this to my toolbelt?
Yes; but not yet. Right now if you want you can use yjs or sharedb. But the APIs aren't spectacular. Eventually I want these things integrated into svelte / react / swiftui and postgres / sqlite / whatever so it just sort of all works together and we get nice tutorials taking you through what you need to know.
We aren't there yet. Automerge is slow (but this is being worked on). Yjs is hard to use well with databases, and its missing some features. We'll get there. The point of all the mathematical formalisms is that if we do it right, it should just work and you shouldn't have to understand the internals to use it.
I'm sharing the vision with you that things should just work. It literally takes you two lines of code to make any of the supported editors collaborative. The Yjs types are a minimal abstraction over the crdt model. I hope that other people build more abstractions around Yjs (for example a react store that simply syncs).
The thing is.. all of what you are talking about is already possible. A wasm port will eventually bring some performance improvements. But Yjs is already very close to optimal performance and it works in all browsers.
But I really am interested in more feature request. I'm mainly interested in building collaborative apps with CRDTs. CRDTs as a data model is an entirely new concept and we need to get experience on how to build apps with CRDTs. And also we need to find out what is still missing.
So head over to discuss.yjs.dev and share your experience.
- Support for moving objects (as well as inserting and deleting)
- Better support for long lived documents. (I want to be able to pull historical deleted stuff out of a document so it doesn't grow unbounded).
- Presence information and markers (this is sort of supported now?)
- The gorgeous automerge API frontend ported across
- Support for interacting with yjs from rust / C / swift / etc
I want all of that too. Although I would add Automerge's API on-top of Yjs as an additional layer. It really does play nicely with Web Frameworks like react. Although for building editor bindings I would choose the Yjs types because they allow faster access to the data. Yjs only provides a minimal abstraction around the CRDT. I hope that other people will build nice stores/abstractions around Yjs that make interacting with shared data easier.
- Presence information and markers (this is sort of supported now?)
Presence is handled by a different CRDT (the Awareness object in y-protocols). All providers implement the Awareness protocol. In many cases you want one awareness instance for several documents, so it makes sense to separate them.
Markers is probably what I refer to as relative positions: references to specific positions in the document. It even plays nicely with undo-redo.
- Better support for long lived documents. (I want to be able to pull historical deleted stuff out of a document so it doesn't grow unbounded).
Yjs does this reasonably well. To a point that you can even build 3d applications like relm.us with Yjs. But you are right that more work could be invested in researching even better solutions.
- Support for moving objects (as well as inserting and deleting)
Moving of elements can already be modeled with Yjs [1]. Although the approaches described are not very convenient to use. Automerge implements this feature natively, but I'm a bit hesitant to implement the next best approach because I see many downsides that conflicts with features I plan to implement. There is a discussion thread where I outline a few of the tradeoffs [1].
Based on everything I've seen about them, OT/CRDTs sound like they are that.
I recommend implementing them for the fun of it. I love it. But don't put anything you write into production until you can leave a randomized test suite running overnight without errors.
Example:
Random op generator: https://github.com/ottypes/text-unicode/blob/master/test/gen...
Which powers this fuzzer: https://github.com/ottypes/fuzzer/blob/master/lib/index.js
You just replied don't. Have you considered he wants to this regardless of not applying them purely commercially and wants to perform research, or in a novel industrial capacity?
> It's just... sad to know enough that I produce less than optimal work but don't know enough to confidently prevent it from happening over and over and over again.
Basically, the notion that not knowing how to implement this means the resulting product is sub-optimal. My counter-argument is that delegating work that needs expert domain knowledge and skills to the experts and use the fruits of their labor will result in a better product.
[1] https://blog.cloudflare.com/introducing-workers-durable-obje...
Do you want a brand name? Go work at a FAANG. Trust me, they're hiring. You'll get there and like me you'll realize they too have their problems and maybe the "over-paid Rails shit solutions" you "know will eventually leak" maybe weren't so bad after all.
Do you want to level up as an engineer? Go work at a hyper-growth startup (probably series B) that is falling over from its scale problems. You will probably end up solving some oddly challenging and novel engineering problem along the way.
Do you actually not mind this stuff but hate feeling like you have to always keep up with the Jones and it gives you anxiety? Go find a therapist that you have good chemistry with and see if you can't work out why you feel this way and how to fix it. To be honest, the corporate rat race thrives at making people who are otherwise doing quite well with their career progression feel like they're not because that makes them easier to exploit. Life is not an exam to be min-maxed. You don't take that fancy career or lifetime earnings with you when you die, so unless you're doing it for the intrinsic joy of doing it, there's an argument to be made that becoming better at engineering has no guarantee of making your life better.
See this blog post for just what it is -- a breathless discovery of a hammer and an author who now wants to use it for everything but may not be at the point where they realize it might not be a great tool for everything. You the person are probably a lot more competent and capable than you're letting on here. And if that's the case and you're unhappy for reasons in your control, hone in on why that is and see if you can't make life moves to change it. There's never been a better time to make a career move.
Good luck. I believe in you.
To be honest, the corporate rat race thrives at making people who are otherwise doing quite well with their career progression feel like they're not because that makes them easier to exploit. Life is not an exam to be min-maxed. You don't take that fancy career or lifetime earnings with you when you die, so unless you're doing it for the intrinsic joy of doing it, there's an argument to be made that becoming better at engineering has no guarantee of making your life better.
Now that I have accepted that my lasting happiness will not be found in a prestigious job or fame as an engineer, where shall I find it? Perhaps the same place other humans have for the last ten thousand years. In the center of a quiet meadow in the woods. Next to an animal that considers me a friend. In the arms of someone I love. Plucking strings on an instrument, singing my stories.
Is this response a coping mechanism, or an acting choice to minimally engage with the parts of human existence which entail naked power brokering and empire building? Maybe it's a little of both. May we both find our own peace somehow, friend.
In this article I explain the basics of the simplest CRDTS the way a web developer without a deep knowledge of CS could relate to it: https://statagroup.com/articles/editing-shared-resources-crd... . I don't present the concept 100% fully, but I give a starting point for someone who "only knows rails". I found most online tutorials dive directly into the theory.
I included all the resources I used to learn the concept on the page too.
It's part of a series I wrote about editing shared resources ( https://statagroup.com/articles/editing-shared-resources ).
I think one thing I'm learning is where to spend my effort. Its definitely a balance, but I'm putting more time/passion into my personal projects/learning, whereas before I saw more opportunities for overlap with my passion and work. When your work doesn't recognize that effort or actively fights against those efforts, it's draining and frustrating.
I've also been thinking about dedicating some serious time to learning. I keep thinking about the idea of taking a month or two off from work. Just treat myself to my own mini-semester of homeschool. I'm somewhat confident I could learn enough to land a better job, but more importantly, one I enjoy more.
I am an under-educated over-paid GED holding Glorified Sysadmin, and while a lot of the shit solutions I work on will eventually leak, I do spend a non-trivial amount of time looking for better solutions. I also try to keep my ears open for what other people are working on and read up on concepts I'm unfamiliar with, whether they crop up on HN or through work.
I learned about CDRTs several years ago at a former employer where a team mate was talking about how it was the future. However, we didn't end up using them. It turns out the more advanced your system gets, the more work goes into it, and the more of a pain in the ass it becomes to run [until it reaches a certain level of operational stability]. So I wouldn't say that using the most bleeding edge research is a good idea even if you know about it. Most people are still fucking up the basic stuff we were doing 10 years ago, and the neckbeards here probably have loads of stories about how much simpler and more stable distributed systems used to be.
HN is a decent place to start. Now that you've read this article, you're aware of the general landscape of concurrent document editing algorithms.
Ultimately, all you really need is the language to a) describe your problem and b) compare potential solutions. I have a CS degree, and it's not like we spent semesters learning every algorithm under the sun. But what we did learn was the language to describe and compare the performance of algorithms (computational complexity theory). Beyond that, it's just about knowing enough terminology to be able to describe your problem, which you could learn just by skimming through an algorithms textbook (or random Wikipedia articles, honestly).
For example: a friend of mine was recently trying to devise an algorithm for ordering buff applications for a collectible card game. Once I was able to find the right language to describe the problem (topological sorting), finding an algorithm to use was easy.
Extra points for buying a dead tree copy and reading it without a thousand alerts and internet temptations vying for your attention :)
I already knew about them, but there will be a stack of other stuff I don't know about. Just get Googling when you hit different problem areas.
I found out about CRDTs because I was working in a niche where a security device was used that breaks TCP between two hosting regions for an application. So I searched for issues to do with that and refreshed myself on the two general's problem, Byzantine fault tolerance, 2 and 3 phase commits and at some point OTs and CRDTs popped up. You'll find this stuff if you keep looking at a problem for long enough.
That was a while ago though and despite still working in that area I've still not implemented anything with them because I'm a contractor and I'm under the direction of people who don't have CRDTs in their toolbelt and don't want ideas from others! I guess my cheerful message is keep at it but know you can only do so much and there's plenty of other pits of despair to fall into even when you're out of the current one. Yay!
And you're not a "shitty software engineer" if you are self-aware enough to know what you don't know. Read, start with Wikipedia, look up the references, play with stuff, explore.
If you need to understand certain things at a deeper level, then explore the classes and other guided tuition.
I've been doing this stuff for 30 years and I still learn every day. You will produce less than optimal work for ever. As a master craftsman (which is what you are, not an "engineer") you are always learning and improving your craft.
You can be satisfied when completing some work if you did your best to produce it and it works within the constraints you had. There will always be things you would do different or better, so do that next time.
Get your intuition down really good and 10+ years is not required.
I might suggest Martin Kleppman's Making Sense of Stream Processing freely available from Confluent [0].
Learn functional programming, gradually. You'll get there. I like to think I reverted a decade of brain damage inflicted by enterprise java development in three years.
Turns your front end app multiplayer with a library.
They manage the hard part and give you a simple API to get the benefits of realtime/syncing/diffing.
Perhaps it would be prudent to read the actual article before tossing off this sort of comment since it's indeed the whole point?
One of the classic problems was size and performance. However both problems have now disappeared. Operations are `O(log(n))` and the size overhead is a factor of 1.5-2.0. That's quite workable.
There are so many resources available to learn CS today. Are you really baffled about how to learn outside of school? Read. Practice. Repeat.
I'd like to have things like CRDTs under my radar to pull from when architecting technical solutions. But I don't.
I have been coddled by the bullshit of "just-good-enough" web development for nearly a decade, and I feel like I will be haunted by it forever at this point.
I WANT to employ mathematically proven solutions, but not only has it been 10+ years since studying computer science (and with it, the relevant mathematics) but I never even graduated.
So here I am, another under-educated over-paid GED holding Rails engineer putting together shit solutions I know will eventually leak.
So, having said all of that, is there hope for adding things like this to my toolbelt? And where do I even start? I feel like going back to school full-time is out of the question, because I have a life now. Maybe online classes? I'm curious to hear thoughts from people who maybe came from a similar pit of despair, or otherwise understand what I am talking about.
It's just... sad to know enough that I produce less than optimal work but don't know enough to confidently prevent it from happening over and over and over again. Is this just the profession I have locked myself into?