It's all fun and games until you realize DynamoDB works more or less the same way: https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
Depends, I’ve known people who have gone through similar experiences and still poo-poo all those “unnecessarily bloated” solutions like a proper database.
we ended up with a lot of shitty solutions to problems that were hard to maintain, hard to extend, and hard to use because the more "complex" solution was really just a fancy version of a folder and some text files.
Just use SQLite, people. Even JSON-in-SQLite is still likely to be an improvement.
I'm not sure i understand how this can happen... unless you try to update JSON in-place (which is a very bad idea for any text-based format), what you do is encode/write the entire JSON from scratch. So either the file is written properly or it isn't written.
Honestly from the entire message it doesn't sound like JSON was a bad idea but that your coworker didn't know what he was doing and if he was doing something else then he'd still be doing big mistakes.
> if there's a crash during the write
It never brings up how can there be a crash in the first place (also the entire article is too filesystem specific).
It's impossible to make crash-free systems. If your goal is to maximize the chance that your data remains valid, you have to plan on that.
2. Lightning struck the 12 V feed and upped the voltage to 10 MV, turning all 0 and 1’s into 6’s
3. Someone spilled a New England Pale Ale on the server
4. The process was assinated by the mysterious killer only known from his modus operandi of leaving OOM written in blood across the syslog
5. Birds nested within the server and fed all the SATA cables to their babies
Seriously though, disk writes aren’t atomic.
1. Write your updates to a copy of the file.
2. Do an atomic rename of that copy to the original.
His sources seem to disagree, certainly with that kind of blanket statement. For example:
Our study takes a pessimistic view of file-system behavior; for example, we even consider the case where renames are not atomic on a system crash.
So this is clearly considered an outlier/unusual.
(e.g., a single 512-byte write or file rename operation are guaranteed to be atomic by many current file systems when running on a hard-disk drive)
[https://www.usenix.org/system/files/conference/osdi14/osdi14...]
I remember reading quite a bit about the (performance reducing) lengths filesystems go to in order to ensure consistency of directory entries even in case of a crash, and for example how "soft updates" were introduced to accomplish the same consistency with less of a performance degradation.
Looking at it from another angle, if you are running on top of a filesystem that cannot keep itself consistent, then you are SOL, there really isn't anything you can do to mitigate.
Just like we can't guarantee that we will be able to persist data that's in memory to disk if the OS is free to kill us at any time. "Best effort" it is, which means getting the data to disk as quickly as possible and not corrupting what is there.
The article linked above never explains that part, it only assumes that it will happen. From the code it sounds as if the crash can happen in the OS itself (but then the entire kernel will crash). At that point things are completely outside your control and you might as well running on broken hardware.
1a. fsync() the file to ensure the contents are durable.
2. Do an atomic rename of that copy to the original.
2a. fsync() the directory to ensure the rename is durable.
Had he pushed further.....
Had he pushed further, he'd have raised funding for the newly invented NoSQL DB, and built a startup company on top of it.
I've been building an open-source alternative on mobile that based on similar concept (SQLite + FlatBuffers): https://dflat.io/ SQLite own schema is already awesome, but in this way, you can have sum-types, better schema upgrade guarantees, index building can be asynchronously etc.
The is the mentality that plagues the industry, that anything more than a few years old is obsolete, and therefore experience is worthless, and therefore the wheel must be reinvented every time because those old programmers must have been dumb, why would they use SQL otherwise. Why real engineers don't take "software engineers" very seriously (and in turn why software engineers don't take webdevs seriously).
Worked like a charm, never had a problem with it.
It was actually put in as a placeholder until we had time to think about a real storage solution, but it turned out we never needed anything more sophisticated, and were actually the fastest and most reliable clients we had. In fact, every time I encountered a performance problem I was hopeful that I would finally have a good reason to do that real implementation, but it invariably turned out to be a simple bug.
- Cocoa has -writeToFile:atomically:, which writes a new file and then renames, so no write-corruption
- We were lucky that lists had just the right granularity for a single file to be read/written atomically
- We likely wrote (quite) a bit more data than absolutely necessary, but I/O tends to have large fixed overheads so medium files tend to take around the same time as small files
- We did not do anything with the data on disk except read it, so not a DB
- We really did use files, not JSON strings inside SQLite
- We flushed to disk asynchronously, but as quickly as possible
But the real answer is that our team was very siloed. No one knew what anyone else was doing. The other problem was that he was actually solving real world problems, and he was a very high performer. He got stuff done. Arguing to start over a project that's already working is a difficult position to hold when talking to management.
The problems he encountered with his dumbass solution were EASILY foreseen by an even noob coder. What did he "get done"? How did writing his own shitty version of a database add value to the company? He is good at finishing his own pointless tasks quickly, maybe, but if I was in charge of the team he would be looking for a new job after this stunt.
Sick-to-death of these cowboys. Nothing is ever "done", the majority of expense in software development comes in during maintenance, not during initial implementation.
Given the team dynamics and lack of involvement from this person's manager, I wouldn't move to fire them. I'd move to rethink the entire team, admonish the manager, and possibly remove them. The team itself wasn't working, and this was a symptom: someone had a bad idea, pursued it for too long, nobody did enough to stop it, and then they couldn't go back.
This is a classic consequence of a manager who has stopped paying attention to their own team. The team was most likely also overburdened with too many tasks, which is why everyone was working on something separate and independent and nobody knew what anyone else was doing. In reality this developer shouldn't have been given a project like this without being paired with a more senior engineer to supervise it, but that would cut down on the number of story points the team could get through and would thus be discouraged in a dysfunctional environment.
As OP said they were siloed from each other and he definitely needed some mentorship. I've seen devs like that turn to incredible coders just after a couple of months of pair programming.
That is 100% a pet peeve of mine: Places that hire perfectly capable jr. engineers and then fail to give them the support they need.
Really? At the expense of everyone else who has to deal with this monstrosity for the foreseeable future, or worse yet replace it with an actual tool that can be reliably used.
This JSON-inside-sqlite-inside-JSON-inside-a-JSON-string beast should never have seen the light of day.
You're not paid to be entertained, sorry. You're paid to be productive. As productive as you can, and to put the needs of the client and the long-term success of the company hopefully first but certainly before any resemblance of entertainment if you're getting paid
Did I mention you're getting paid to work?
If it's any consolation, it fell on to me to maintain this code after he moved on to something else, which is why I know so much about how it works.
This has happened to be before. I disagreed with a technical direction, it was implemented anyways, and then I'm left to maintain it. Very frustrating.
I've dealt with this before, it is a form of gas-lighting. Some people are good at making everyone else into a bully when THEY are the actual bully. Like the kid who keeps splashing you in a pool, but runs off and cries and tells when you splash them back.
Standing up for yourself sometimes makes you look/feel like an asshole. That doesn't mean you are wrong or that you shouldn't do it.
edit: Check out the book "Radical Candor" if you regularly struggle with expressing negative feedback
That's not any consolation... If anything, it's all the more reason for you to be pissed off. He should have dropped the project, rolled out a future-proof tool and taught to do differently next time.
Anything short of that is just enabling the dude's delusion of grandeur and therefore a mistake on everyone else's part...
Just like the never ending "turn this Excel workbook into an app" stream of work, refactoring older apps will be a constant. Focusing today's conversations on yesterday's mistakes only detracts from the work left to do (which is to say if your architecture change arguments are valid, there should be ways to justify implementing them today outside of "it should've been done this way in the first place because then we wouldn't have had those problems that are now solved anyway")
I'm pretty sure the cascading series of "his next problem" sentences implies that there were plenty of problems with the architecture that weren't identified ahead of time, and they had to encounter and then fix as a series of bugs.
> And still, there's a lot to be said for keeping your developer's entertained so they stick around.
There's a difference between keeping your developers entertained and letting them infect production with ill-conceived projects that cause problems for all those that interact with them.
This project is reimplmenting something already solved multiple times. There are many document stores, and JSON interfaces and addon to traditional RDBMS, so what was being solved here, other than letting someone scratch an itch at the expense of the division he's working in. You're better off giving him 20% time for his own projects and calling it a day if you really think entertaining your developers is important enough to warrant it.
There are times when rolling your own is useful. Generally when there's some extreme requirements for space or performance, but even that becomes rare when the area is mature and explored thoroughly. A database, even a JSON document store of some sort, is so mature that to make it worth while for one person to roll their own when it seems to need all the common features (locking, remote access, different clients), that to actually recoup the cost of building our own (much less the future cost of troubleshooting and bug fixing) is almost impossible unless you're somehow hired a genius workaholic for peanuts.
Once you've abstacted it to a service, your API is what you and your client (should) care about, and many of the arguments for more specialized implementations no longer apply. Personally, I think the only reason I would go with something like sqlite instead of Postgres/Mysql behind a microservice is if I was baking the date into it with each release, so the sqlite data files are shipped with the version released. Even then, I'm not sure there's any reason I would do anything other than sqlite though. Even if I had need of lots of JSON files, I would probably have my build procedure process them into an sqlite file I tested and shipped with, if only because I would then avoid having to deal with all the problems this guy encountered by trying to make his own database.
< "Well all the work was already developed, and it would take too much time to rewrite it. You should have said something earlier" > "When?" I asked, considering she had just put up the (big) PR's and PR's ARE the time to review... < "Check her commits as she pushes them to the repo" - as in her bugfix/feature branches, not master...
My jaw dropped. Especially since I was hired on as "Lead" and had all the accountability but no actual power.
It's incredibly frustrating because during code reviews I will request changes so it's not such a broken hack job, and the response will basically be "No, it's not worth changing". At which point I'm the one "holding up development". We wasted hundreds of development hours during the last project because of this persons "inventive" code, and nobody seems to understand what's going on.
Shame the job market is a bit crap right now.
Two pieces of actual useful advice I can offer are:
1. A review style I picked up based off of RFC 2119[1] basically the reviewing software we use allows us to mark particular comments as blocking of non-blocking and I pair that with the usage of MAY/SHOULD/MUST within the comment language i.e. "We're using the old `array()` syntax here instead of `[]` we MAY wish to use the more modern syntax" this allows me some room to elevate necessary change while keeping in the nitpicks I really want to throw in (and I do try and minimize them) without lowering the power of the strong comments. I've used MUST maybe three times always for something incredibly terrible like pages not loading or migrations to the DB that are unsafe and cause data loss.
2. Agree on syntax and style rules and enforce them. It's easier to get people to agree to rules once than try and argue for them on each PR - anything like brace placement or line limit shouldn't come up repeatedly since it wastes everyone time and makes folks feel belittled.
Post-mortems are great for many reasons. For the case of GP, one particular advantage is that they align senior peoples' understanding: we shouldn't do X again. If you have a strong narrative for why a project failed, post-mortems are a formal setting in which you can present this narrative with concrete evidence to higher-ups.
In the future, when you see warning signs that a mistake is approaching repetition, you can raise the concern up the chain, invoking the memory of the post-mortem to motivate their intervention.
I also totally agree that a sincere and high-quality code review process is required for high quality code. Your 2119 recommendation is excellent. I'd also recommend doing some reading on commit message templates that smart people follow, they've improved my commit game, big-time.
We've had that process on for quite a while, and while there are some big weaknesses and holes in it we've also adopted a principle to keep PRs as small as possible[1] with those two tools we've had some pretty reasonable success with a lot of our biggest incidents being related to times when we've made large changes or a review was skimped on.
1. Even if that isn't measure in LoC - moving a dependency and updating references to it is something I'd count as a single action - but one I'd want isolated from any logic changes.
The commits I was told to review if I wanted to stop thecraziness were the personal ones going to the bugfix/feature branch.
Many good companies enforce a no-origin-branches policy, with rare and well-justified exceptions. Because, used as you describe, a "feature branch" is just a future massive diff in disguise (when it's eventually merged), and massive diffs are a big no-no because they're a huge pain to iterate on via code review.
The workflow looks something like this.
git pull master; git checkout -b my-feature; ... ; git add -A; git commit
At this point, you submit the code for review, and upon approval the branch is merged into master and pushed. It’s not possible to push a commit hash to master that has not been reviewed.
If you have a feature that’s composed of many steps, you can “stack” multiple commits, and review/merge them in order.
If you want to develop the entire stack at once, you’re most likely doing something wrong (according to this culture). You can incrementally merge pieces of code to master in such a way that’s impossible for it to be deployed, and your final diff can be what makes it deployable.
Encouraging smaller changes isn't nearly as useful if those changes aren't isolated - if it's just half the picture then you can't accurately review it.
I hit a similar sort of issue recently - I've been incrementally developing a complex data migration, each change to the migration has worked on its own and been reviewed separately but I'm still going to go in and request a full review once the piece of logic is fully assembled. This is also happening on an integration branch on origin - we do try and keep these to a minimum but we're making a backwards incompatible change that would be quite expensive to do in a fully backwards compatible manner.
There are things that are infeasible to reasonably do without an integration branch (nothing is impossible technically, but it might be a huge waste of time) but even those things are pretty few and far between. If integration branches are common place at your company it might be good to examine coding practices and see if you can slice up tickets to be smaller.
But, I do believe it pays off in the form of a higher quality end-product (fewer bugs, more testable/legible components, more extensible), which saves you time in the long run.
This sort of stuff is what deters me from being a developer sometimes. Fuck the salary, get me out of here.
The GateKeeper process isn't something you want to index on too heavily - but you also need a mechanism to counter-balance the possibility of a dev saying "I built a prototype last week that does 95% of the things we want" and 3 months of iteration later identifying that it only did 5%, and that getting the remaining use cases will require a re-write.
https://www.cs.ait.ac.th/~on/O/oreilly/perl/cookbook/ch07_09...
https://github.com/skorokithakis/goatfish/
It's actually quite good as a quick-and-dirty datastore. I do need to move everything to SQLite's JSON field, though.
My question is, why didn't he go for any of the existing solutions when setting them up would've still been faster than rolling his own DB-in-a-JSON-file solution?
Only downside is you need to format ext4 with type small otherwise you run out of inodes before you run out of disk!
So I built my own database in PHP.
Enough said
Like, initially allocating a large file block, and then you subdivided that yourself to get individual sector access?
I imagine if this database system is contained well enough, it shouldn't be so difficult to swap its internals with something else. Especially if it's all just JSON-like.
You can have structured relational databases (RethinkDB for one, but there are others)
The first problem he encountered was that multiple connections couldn't both be using the database at a time without clobbering each other. "No problem," he thought, this is a good use case for micro services. A service sitting on top would ensure that there was only one operation being performed at a time.
Next, his problem was that the database would get corrupt sometimes when something bad happened in the middle of writing the file. His solution was to put the entire JSON format inside of a JSON string. If it could be parsed successfully, then he knew the whole file was written. Then all he needed were "backup" files for each table, in case the current one was corrupt.
Next, his problem was that querying and iterating through a large table performed badly, since it required parsing the entire thing first. Querying several times required the whole file to be parsed every time. The solution was to move SOME of the tables over to JSON-inside-SQLite.
EDIT: Oh yeah, the next problem was how to structure the data inside of sqlite. He decided to make a single table called "kitchen_sink" that held every JSON value. There was a column that said which "collection" it belonged to. There was another column that represented the row's primary key. So you could quickly query for a collection name, and a primary key, and get the full JSON row.
So the next problem was that you couldn't query quickly for things that weren't the primary key. So new columns had to be added called "opt_key1" and "opt_key2" where certain rows could put key values, and indexes could be added on those columns, so you could quickly query by it's first optional key, or it's second optional key.