Or we trust our knowledge of such solutions more than even your (anyone's) coding abilities :)
I realize, of course, that you're using "trust" as a metaphor for expertise, as well as that we all have a tendency to be biased in favor of solutions that favor that expertise/familiarity.
The trouble is, programmers outnumber ops people by a vast, vast margin, especially here on HN.
> often, an ops person can take your developed software as a black box, and solve its scaling problems without touching the box.
Not often enough, not any more. It seems that this skillset has been sufficiently devalued in the industry that, however rare it was originally, it hasn't spread, which means there may well be a significant plurality of coders out there who have never worked with such an ops person.
> And this is often the optimal way to solve these problems: you probably can't write a caching layer for your web app, inside your web-app's process, that will work half as well as Memcached
This goes back to my original, partly facetious, comment about trusting anyone's coding. My real point is that I don't need any kind of faith in something like memcache, since I've seen it work many times before, whereas someone's coding ability (even my own) does require a leap of faith, since it's totally unknown how effective the effort will be or on what time scale.
> And, since these scaling problems can be solved without ever touching your code, it's especially silly to try to anticipate scaling problems you might have and solve them in your code
That's a message that's difficult to hear for management that's made up entirely of programmers (such as at many software startups, even grown to a larger size). Witness the willingness to pay huge premiums for cloud ("infrastructure as code"!) solutions, that eliminates huge low-level scaling opportunities.
No. In this case, the metadata included some of that data, but it also stored information about past application state or data that was otherwise stored in the blob itself.
For instance, when a file was replaced, it created an orphaned blob in S3. Tracking that down meant using rollbacks in Heroku postgres on at least one occasion. Keeping track of which files replaced other files is new functionality that was not possible using any reasonable means, but would have come for free with the feature I designed.
Reading data out of S3 is painfully slow and I don't consider it to be a reasonable solution for anything that involves looking at more than one file. Consider the question "how many M4A files have a Content-Type set that isn't M4A?" Iterating hundreds of thousands of files and checking their magic number (ignoring ID3) and Content-Type is not only slow but also expensive. Storing the content type in advance would at least allow that process to be shaved down to a small fraction of what it otherwise would be. (and before you ask, file extension is a worthless heuristic when dealing with user-supplied data).
But of course, I didn't know that storing an object with metadata would solve these problems, even though I knew that I should do it.
Saying things like "put Varnish in front of S3" is suggesting putting your finger in the dike. The cost of setting up and maintaining Varnish (and paying for it) is far higher than just doing the thing the first time. And in fact, would have been far more costly than me putting my foot down and doing the one-time migration, as I did. Besides, our applications are on Heroku, where something like this would have been non-trivial anyway.
My point is that YAGNI purism optimizes for short term wins. If you never do anything until you can make a compelling argument that it's something you NEED to do NOW, you're going to end up with Dr Seuss-esque systems that do everything possible to actually avoid solving the problem itself, especially if there's always a (less than ideal) workaround. This is one of the easiest ways for tech debt to accumulate.
Are you sure? What is your time worth?
I mean, there's certainly a comparative-advantage thing here. A software developer setting up a Varnish instance themselves is going to take more time than a software developer writing code, because software developers know code and not Varnish.
But my point was that most software isn't run by single do-everything DevOps people. In bigcorp production-scale systems (where all this software engineering advice both comes from and is targeted toward), you've got a dev team creating a particular software component, and then either internal or external ops teams running it as just one component of their system/solution. Those ops people know Varnish better than they know code.
Picture a ticket triage that goes through a support person, and then gets elevated to an ops person. (Because this is what ops people are for, whereas the dev-team's time is far too valuable to the company to spend on things that could be handled by the pre-provisioned capacity of the ops team.) The ops person isn't going to turn around and ask a dev to solve the problem by writing code if he can at-all help it. The ops person is going to try to solve the problem themselves, with an infrastructure-level solution. And, given
1. the overhead of having to go to the dev-team and get the ops-supporting feature added into their priority list, and then work with the support person to interact with the customer while the dev-team maybe eventually fixes the problem (and where then the ops-team will have to both deploy, and understand how to maintain, the software in its newly stretched state, where it probably now has extra ops-time needs like cache storage!);
vs.
2. the ease with which existing deployments of infrastructure components like Varnish, which were stood up to solve previous scaling challenges, can simply have their cluster-configurations extended to support new scaling challenges with no marginal increased maintenance burden;
the infrastructure-level solution will win every time.
Yes, sure, if you're one dev and you do your own ops and you don't actually know much about ops (in the comparative-advantage sense), then solving all your problems in code might have the highest ROI.
YAGNI—and pretty much any other software-engineering principle—isn't targeted at you. You're doing artisanal software development—making the hand-woven wicker chairs of the software world. Most such principles focus on decreasing the Total Cost of Ownership of software, taking into account inter-departmental collaboration overheads, maintenance of the codebase after the original developers leave, etc. You've got none of those concerns.
If you're a dev-team of one and you deploy your own code, go wild: write your whole system in a macro-heavy DSL dialect of Common Lisp with your own custom logging; or write your system as a single 46kb x86-64 assembler unikernel. It doesn't matter, because you still understand it, and can modify it just fine. (Just don't expect to sell your company down the line!)
---
> Consider the question "how many M4A files have a Content-Type set that isn't M4A?" Iterating hundreds of thousands of files and checking their magic number (ignoring ID3) and Content-Type is not only slow but also expensive. Storing the content type in advance would at least allow that process to be shaved down to a small fraction of what it otherwise would be. (and before you ask, file extension is a worthless heuristic when dealing with user-supplied data).
Ah, yeah, that's a different problem than I had assumed. In my mental model, you already had a proxy for handling your object creation, where audio files users uploaded would first go to you; you'd extract the file's indexable metadata (magic, ID3); and then your server would add said metadata into the S3 PUT request as headers (Content-Type & co, and then x-amz-meta- headers for anything else.)
Under that model, you get back everything you need from just doing a HEAD request to your object. S3 HEAD requests are cheap. They're just not fast/highly concurrent, so the point of Varnish here is to make them so.
S3 is annoying, though, in that you can't mutate metadata on an object without re-uploading the object. So, if you were allowing people to do direct object uploads to S3 using signed URLs, and then doing the indexing from there, it makes sense that you wouldn't want to write the resulting metadata back to the object, and would instead want to keep the canonical copy of it local.
In that case, though, there's still an infrastructure-level solution. Replace S3 with https://www.minio.io backed by S3, with Minio's metadata storage pointed at your existing RDBMS. It's essentially just the same thing you built, without writing code. Metadata lives in Minio (and therefore your DB, where you can query it); object bodies get uploaded asynchronously to S3. Though, with this solution, you can optionally have the metadata get proxied through Minio during the PUT, and thus do the metadata-generating content analysis step asynchronously against that stream (as a Minio plugin), rather than having to wait for the upload to complete and then GET the result back to your business layer.
Though, again, for your devs=1 use-case, maybe this has higher TCO than just writing code to track metadata within your system.
If so, I hate to say it, but doesn't YAGNI still apply here?
This was, essentially, a scaling problem: the O(N) time-cost of querying metadata for N S3 objects was too high. You still could have written code to query that metadata out the "naive" way anyway, and it would have worked for low N. But it wouldn't have worked "at scale."
But there are two [point five] ways to solve a scaling problem:
1. When you need to scale, write more code, customizing your logic to make it more performant, add locality or caching, etc. (This is what you did, and the way most software engineers think.)
1.5. Anticipate the need to scale, and write code "the more performant way" in advance. (This is where the YAGNI admonishment comes from.)
2. When you need to scale, attempt to find an infrastructure-level solution that involves writing no code. This is the way ops people tend to think, since they don't write code (or at least, they trust their knowledge of infrastructure solutions better than their coding abilities.)
An example of approach #2, in this case, would be something like "put Varnish between you and S3, and configure it to only cache HEAD requests (and to synthesize HEAD response cache entries from proxied GET requests, without caching the GET response itself.)" Then your existing O(N) S3-metadata-querying code would—after warming the cache—suddenly be faster; it would probably be fast enough to answer whatever sort of debugging questions you'd like.
The reason people say "YAGNI" is that, often, an ops person can take your developed software as a black box, and solve its scaling problems without touching the box. And this is often the optimal way to solve these problems: you probably can't write a caching layer for your web app, inside your web-app's process, that will work half as well as Memcached. Or a logging system that will work half as well as rsyslog. Or a web server that will work half as well as Nginx. Neither can the people who develop packages for your programming language's ecosystem. The real experts in "what you need for production-scale" converge on an infrastructure component and develop that, rather than contributing to FooLang's logging implementation.
And, since these scaling problems can be solved without ever touching your code, it's especially silly to try to anticipate scaling problems you might have and solve them in your code, early. The only scaling problems you should worry about at design time are the ones that can't be solved by IPCing infrastructure components together. (Such as, for example, the cost of a huge number of concurrent threads. Erlang/Akka/etc. exist because they solve a particular scaling problem that is intractable to anything but a process-architectural solution.)