To the forum's credit, the top comment (by u/judofyr) the last time tailscale blogged about their unusal db setup, got it exactly right: https://news.ycombinator.com/item?id=25768042
> Interesting choice of technology, but [tailscale] didn't completely convince me to why this is better than just using SQLite or PostgreSQL with a lagging replica.
> In particular [tailscale has] designed a very complicated system: Operationally you need an etcd cluster and a tailetc cluster. Code-wise you now have to maintain your own transaction-aware caching layer on top of etcd. That's quite a brave task considering how many databases fail at Jepsen.
> Considering that high-availability was not a requirement and that the main problem with the previous solution was performance ("writes went from nearly a second (sometimes worse!) to milliseconds") it looks like a simple server with SQLite + some indexes could have gotten [tailscale] quite far.
> Wouldn't also be HN without the misinterpretation of BrandonM's response. Here is dang's comment about that topic/meme: https://news.ycombinator.com/item?id=29178442
Its different reading it at the time because it factors in the context of the community at the time as well as the lived experience of what it was like at the time to be a member or participant in given discussions. With that contrast, you can then get an idea of the intent of those comments beyond just the words, as they are often (and in this case) reinterpreted through the lens of current day contexts.
Example: ask what people think of the hit TV show The Apprentice today in the present, then hop in a delorean and ask someone before the trump presidency or candidacy what they think of the show.
You can call me anytime <3
I think the underlying actual question for Dropbox to ponder (but perhaps not answer publicly) was "what is your moat?" without the business development language and at least for me that remains an open question.
I remember I was in college when Dropbox started and I thought Dropbox was silly to offer more storage for inviting more users. It was easy to approach people and ask them to try something in college. I think I almost maxed out my invites. I remember feeling I was taking Dropbox to the cleaners...
One activity question for HN: why do you think Dropbox continues to exist today while copycats like Barracuda Copy have failed?
FWIW, i've been using Dropbox since practically the day they went online, and have been a subscriber for all but the first few months of that time. They've never once failed me, never been offline when i needed them, never lost anything, they created a Linux sync client out of the box (whereas Google first claimed the would make GDrive for Linux but never did), and it's easy to manage via multiple interfaces (mainly their own web UI and https://rclone.org, which can proxy your dropbox behind an add-hoc FTP server running in your LAN).
They're extremely pricey compared to the same storage in GDrive, but Dropbox has never let me down so i'm in no hurry to drop them.
"...Half-a-billion people stored files on Dropbox. Well, sort of. Really, the files were in Amazon’s cloud. .." [1]
[1] "The Epic Story of Dropbox's Exodus From the Amazon Cloud Empire"
https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama...
The challenging part is to provide the back-end. Get massive storage, with proper checksum integrity, at a cost effective value and with flawless security.
A single command (and minor variations around it):
aws s3 sync <name_source_dir> s3://<name_target_bucket>
get's you all that. It can be easily mapped to an intuitive user interface by a software development boutique in weeks.
And because that is the difficult part, it seems for the first 8 years of their history that was solved for them by AWS, according to their own reported history.
Sharing a folder or file from Dropbox is no more intuitive than using the S3 console. Looking at the very colorful history of security issues around Dropbox [1] it's certainly not safer.
So why would a user pay more for storage costs than with a cloud provider, with notoriously less security? Does a tiny layer on top one or more cloud providers justify the add-on value for the user or the current valuation of 9 Billion dollars?
[1] "Criticism of Dropbox": https://en.wikipedia.org/wiki/Criticism_of_Dropbox
side note: I'm filled with regret for every document that I authored on Dropbox Paper. It didn't last for long, but there was a time that it was the official wiki for a company I worked at. Its not impossible to export, but never with full-fidelity.
I've not really used the S3 console but, at least on macOS, sharing a file from Dropbox is as simple as "right-click on the file in Finder, Share..." then enter an email address or create a share link from the popup. Or I can right-click and "Copy Dropbox Link" if the direct link is good enough.
Is the S3 console simpler than a right-click?
aws s3 presign s3://awsexamplebucket/test2.txt --expires-in 300
How is this easier than "right-click, pick a menu option"?
(If I want an expiring Dropbox link, it's there under 'link settings' in the pop-up. Admittedly only has date granularity but still there as an option.)
Not that I like Dropbox too much. I'm going to switch to my self hosted solution. But it's certainly not as easy as using Dropbox, even as someone who uses AWS since 2012 frequently.
This article could have easily been "why we were down for a day while some auto-vacuum setting was broken." Then everyone would be like "why didn't you just scp a json file between machines" ;)
Say "switch databases on a successful project" and I bet people's minds go there. That is indeed expensive and painful, but if you have a small team, it's just being agile.
There is also the "punching up" / sour grapes aspect that can't be ignored. "I would have picked the right database from the start, but nobody's funding my crazy idea." Life just isn't fair, and HN is where you can vent, I guess.
The kinds of projects that I more generally tend to see make these migrations successfully are the ones that design with in-memory data structures first and then sort out how best to persist that kind of thing to disk after their domain model has solidified a little bit.
Tailscale smells a lot like that to me.
But as I’ve been using only statically typed languages the last few years (on the backend), I’ve moved to designing the domains in the code, like you said.
When Tailscale migrates databases, it is a few engineers working for a few weeks solving problems that already exist.
When a company with 50k employees does it, it takes two years and only makes things worse.
My go to language of choice is PHP/MySQL. I've never not been able to scale a project. At one point while working at Comcast I prototyped a system for every error code from every cable set top box (150 million) in the country on a MySQL server instance running on a MacBook Pro pulling data from Splunk in real time. All so I could generate some png charts and embed them in Slack.
I know there are limits to MySQL but so far in my career I haven't had any projects hit those limits because hardware has been getting better faster.
The replies are focused on MySQL (and to a lesser extent, PHP). None of that matters.
The hero here is the logging system serving as seekable stream of collected events. (I’m trying hard here to not say K-fka).
By consuming events from that source and being able to perform time-based seeking, you’ve offloaded (or dare I say, eliminated) your scaling pressures.
Even if you commit the “sins” of no indexes on heavy queries, using MyISAM, or building a giant, memory-intensive associative array… none of it matters.
You can choose to wipe your local dataset, start again in a different language, switch to another data store model, and none of your downstream consumers will know nor care (unless you blog about it).
*This* type of design, that prioritizes consumer-facing resiliency and continuity, is what I look for in systems that scale.
That Comcast prototype sounds amazing. PHP and MySQL can really bring you very far, i use it for nearly everything and it is so incredible easy to develop and “deploy” new versions. I use propel as ORM and it is very close to the DB and I can change schemes fast and just let the autoupdate handle the rest.
A k8s cluster for something like that wouldn't cost anywhere that amount, plus you would get the advantage of being able to deploy your app in multiple availability zones. If you don't need k8s features, there are definitely simpler options that will give you redundancy and won't break the bank.
I have gone past mysql limits a few times. Will you go beyond its capability is a simple measure. Can your hardware write fast enough to keep up? If your hardware maxes out writing to disk 200 megs per second, welp mysql maxes out at 200 megs per second. If you need 400 megs per second of writing, well you need two instances that can write independently. That is when you are fucked. At that point you get into sharding* and doing whacky things mysql isn't meant todo. Reading things from disk, mysql's replication scales just fine.
Later on i met the roomate of the author of memcache. Few hundred lines of C. Completely changed the game how caching systems worked. Years later a super senior FB engineer explained to me over a beer - "we basically use mysql as a file store, everything else is in memcache." Good ideas are stupid simple.
imo they’re not solving a notoriously hard problem, they’re realizing that they have the tiniest version of this problem (and that will most likely continue to be true). when that’s the case and you want to prioritize stuff that moves the needle like dev velocity, why do anything less simple? getting hung up on building scalable solutions far beyond what you need is somewhere between a fool and a big co employees game. far too often people introduce complexity way before it’s needed, and that almost always slows down your devs ability to actually fix bugs and make the product more reliable.
Which is just a piece of cooked fish, rice, and miso soup. The critic is first almost insulted by such a simple meal. But then realizes the fish, rice and soup are prepared to perfection, and is amazed at the chef’s skill.
Similarly, I’ve also heard (probably from a show like Top Chef, so YMMV) things like a salad or soup are also quick ways to differentiate chefs. In this case, it’s less about technique and more about identifying and mixing flavors.
It works because there is no place to hide. You either have good flavor or not.
You mean Ron Jeffries?
Consider whether the secret sauce here might not be au courant database choices, but rather something much harder for random teams to pick up and run with: just straight up good design and programming.
This is a thing that happens in other fields, too; some of the hardest dishes to cook seem extremely simple and have few ingredients. They're hard because there's no place to hide: your technique has to be flawless or they don't work. A famous entrance exam for new cooks at fine dining restaurants is "make an omelet". Tailscale is, thankfully (and, to me, irritatingly), more like an omelet than a truffled hash.