So there has not been any major improvement in MySQL for 3 years? There hasn‘t been released any minor version? Can someone explain the current state more?

No. MySQL 8 changed its engineering approach with features released in minor releases.

In this regard it is like MacOS X or Windows 10, whatever you prefer where features are shipped without major version change

Sure, and this is great. But The newest release is 8.0.26, so there hasn‘t been any minor release yet - in 3 years. And i can‘t find anything about new MySQL features or planned features except the big 8.0 release.

There have been several minor releases in the past 3 years: https://dev.mysql.com/doc/relnotes/mysql/8.0/en/

In semantic versioning parlance those would all be considered "patch" releases, not a "minor" release. The minor version segment in a semantic version is the 2nd number, so most people would consider the next "minor release" for MySQL to be 8.1.0.

If following the guidance set out at https://semver.org, a patch release wouldn't add any new functionality, it would just address bugs in a backwards compatible way. A minor version would introduce new functionality in a way that doesn't violate backwards compatibility.

Quickly scanning the release notes for some of the MySQL 8 versions, they seem to be generally sticking to this; most of the versions fixed some bugs or unintuitive behavior, but it doesn't look like much new functionality has been introduced since 8.0.4, the last release candidate for the major version bump.

MySQL does not follow semantic version and there have been features introduced in minor released including some incompatible changes as you can see for example https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-24....

That's still a patch release not a minor release.

If it does not follow semantic versions, it is just a third-level version number. It's like Linux's 2.6.x series.

Yeah, no thank you. I'll stick with Postgres.

Many major new features have been introduced in 8.0 point releases. Some examples:

* 8.0.12 added INSTANT algorithm for adding new columns to a table

* 8.0.13 added DEFAULT expressions (column default values may now be any arbitrary expression, not just a constant)

* 8.0.16 added CHECK constraints

* 8.0.17 added the CLONE plugin for rapid physical copying of a db

* 8.0.23 added INVISIBLE columns (excluded from SELECT *)

These are just a handful among many others... basically MySQL 8 does not follow SemVer at all, nor does it claim to. (Although personally as a tool developer in the mysql ecosystem, I selfishly wish they did!)

That’s pretty scary, isn’t it?

No, since MySQL 8 doesn't ever claim to use SemVer. They have an explicitly stated policy of including features in point releases. Why would it be "pretty scary" that a non-SemVer software doesn't use SemVer?

SemVer is an arbitrary versioning scheme, not a universal standard. It's the operator's responsibility to understand the versioning scheme prior to upgrading.

It is pretty scary because most of these new features or enhancements are buggy. By not adopting SemVer, users of MySQL who prefer stability (oxymoron?) have no obvious upgrade path. They could be happily using version 8.0.123, and then upgrade to version 8.0.126 to get some security fix, and suddenly encounter a bunch of functionality and/or performance regressions.

Some examples...

WL#10310 Redo log optimization:

Feb 2018, initial commit 6be2fa0bdbba, landed in 8.0.11 (Apr 2018)

Jun 2018, crash regression fix commit 270d18368650, landed in 8.0.13 (Oct 2018)

Apr 2019, performance regression fix commit 75c4f7161a56, landed in 8.0.18 (Oct 2019)

WL#5655 - InnoDB: Separate doublewrite file to ensure atomic writes:

Dec 2019, initial commit ce14ef911, landed in 8.0.20 (Apr 2020)

Feb 2020, data loss regression fix commit c1bc61dc7, landed in 8.0.20 (Apr 2020)

May 2020, stall regression fix commit 00b284707, landed in 8.0.21 (Jul 2020)

In comparison, Cassandra during the DataStax days was extremely buggy, but at least it tried to follow SemVer. So the operators can follow some simple guideline, e.g. >=2.0.14 is fine, >=2.1.17 is fine, >=2.2.9 is fine. Of course regression can still happen, but that would be an exceptional case.

This is a misconception that new features introduce more bugs than eg. patches/minor version upgrades. I can only bring up the OpenJDK project as example but this sort of change happened there as well - what used to be JDK 8.xxx and everyone jumped to that without second thought is the exact same thing as JDK 9->10. It’s just management’s backwards thinking that somehow the former is safe to apply while the latter is not.

There is certainly no hard and fast rule. In the JDK case, in terms of language design, my observation is that with Brian Goetz in charge, it doesn't matter too much whether it was the slow release cycle back in the pre-9 days, or the shorter one from JDK 9 onward -- whenever a new feature is introduced, all kinds of compatibilities are being maintained at all costs.

(Except Project Jigsaw of course, but I'd like to think that he got overruled by his boss)

As for bugs, I will just quote Gil Tene from https://www.theserverside.com/opinion/Dont-ever-put-a-non-Ja...

> Play with them on your laptop, but don't use a single feature, and wait for the LTS.

Personally I think the LTS model is great. Features get rolled out as soon as they are ready, and then get stabilized after being used by early adopters. Meanwhile, production can stay on LTS and continue to get just the necessary fixes.

In my experience, it is quite an exaggeration to claim that "most" post-GA 8.0 features or enhancements are buggy, especially in the general case. Yes, some of them may have edge-case bugs or performance issues, but typically only affecting some users with very specific environmental or workload situations.

Operators must test before upgrading. Luckily some third-party software makes this easier, e.g. Percona's pt-upgrade or ProxySQL's mirroring feature.

I'll state again unambiguously, I'm not personally fond of 8.0's release policy of rolling out new features post-GA. But I view it as an annoyance, and at least one with some upsides (not having to wait 2 years for a massive feature dump all at once) rather than being uniformly negative or "pretty scary". Personally I was careful about testing point release upgrades before 8.0, and I'm still careful about it now.

I’m not saying semver is a universal standard but this is _wildly_ different to everything else.

Nobody expects defaults to change between patch/minor versions.

This is another example of how MySQL does not give a flying F about the footguns it leaves lying around.

Putting the onus on the user is not reasonable; especially in a world where we’re trying to reduce toil or get rid of ops completely.

In another subthread, you're (justifiably) unhappy with how they didn't change the strict-mode default soon enough; but here you're upset that some defaults changed too fast?

Also my comment above in this subthread wasn't even about defaults. I only mentioned some features added, and you responded saying it's scary and then talking about defaults?

You don't like MySQL, I get it. Regardless of reasons, I don't think your view will change, so why enter every HN thread about MySQL just to repetitively bash MySQL? What's the purpose of this?

The purpose, mainly is to warn people of the pain I’ve suffered. My life should be a lesson here because honestly it is _painful_ to realise you have lost monetary data and have to go hat in hand to the CTO and CFO to explain why.

The strict mode setting should have been the default since the beginning. It is absolutely unthinkable that it wasn’t, the only justification I can think of is that:

a) it wasn’t built with that in mind and thus was experimental.

b) was not enabled to ease adoption (this is kinda evil, in my personal opinion because it teaches bad habits)

c) was considered a power-user feature, which is ridiculous.

What you’ve said just exemplifies the trend I’ve seen before: MySQL does not care about creating footguns, and people who have bought into the ecosystem like to exhalt that “you’re holding it wrong”- which is absolutely not a good warning.

If you listen to my rambling about the pain I’ve seen, and you still want to use it: at least it’s an informed decision. But pretending that all is well is not ideal, it doesn’t lead anyone to make better software or practices (MySQL) or better developers (because they don’t see footguns before they’ve sprung some self-inflicted wounds on themselves)

> The strict mode setting should have been the default since the beginning.

I absolutely agree with that. But what's the point in talking about a terrible default that changed nearly 6 years ago, literally over and over again, year after year?

> the only justification I can think of

The real answer is almost certainly "backwards compatibility". Same reason many things in Windows are non-ideal, for example.

Someone made a really bad technical decision early on, but fixing it overnight will break things for an utterly massive number of paying customers, so it can't be rectified without a slow transition plan. It happens.

Yes, it absolutely should have been fixed on a faster schedule regardless, given its importance. But just because it wasn't, doesn't mean that the entirety of the piece of software is hopelessly flawed and everyone involved in its development is an utter cretin. Especially when a piece of software is multiple decades in age and has gone through multiple corporate acquisitions.

> What you’ve said just exemplifies the trend I’ve seen before: MySQL does not care about creating footguns

What I've said? What footguns? I'm asking this honestly: what are you referring to? Let's recap this subthread:

* @taywrobel said "it doesn't look like much new functionality has been introduced since 8.0.4"

* I listed a number of new features introduced in 8.0 point releases, and explained that MySQL 8 is intentionally not doing SemVer

* You said "that's pretty scary" without ever elaborating on which specific thing I said you find scary

* I disagreed, given MySQL's stated written policy of not following SemVer

* You mentioned something about defaults changing (??? this wasn't even a topic I mentioned by that point in the subthread)

Now you're talking about footguns again, but the only one you're mentioning with any specificity was fixed many years ago.

> people who have bought into the ecosystem like to exhalt that “you’re holding it wrong”

Where has anyone said anything remotely like that in this subthread? Are you referring to my statement that the user should understand what's in an upgrade before upgrading? If so, I absolutely stand by that statement; it's utterly crazy to upgrade any database without even glancing at the release notes, let alone having a basic understanding of how the database vendor handles releases and versioning. This is operational basic practices 101, not about "holding it wrong".

And for what it's worth, "people who have bought into the ecosystem" includes a massive chunk of the S&P500 using MySQL as a primary data store. If it's as deficient as you claim, why aren't all these companies going out of business?

> Nobody expects defaults to change between patch/minor versions.

Can you provide an example of an application-side (IOW, not a DB-infrastructure-type variable) that has changed it's default in a patch version after GA in 5.7 or 8.0?

IIRC, all of the defaults changed in very early non-GA versions.

> _wildly_ different to everything else

Linux?

Linux never changes backwards comparability. Linus is literally famous for saying as much.

> Why would it be "pretty scary" that a non-SemVer software doesn't use SemVer?

Is there another level of patches below point releases for backported security patches, or are your options to upgrade to something that adds new features and may be backwards incompatible, leave your system vulnerable to known security wholes, or patch it yourself (or maybe depend on your distro to do itl

In practice, this is rarely a problem unless you're a cloud vendor with a DBaaS product; the most common concern there tends to be bugs that cause server crashes. But even that is fairly rare, and those platforms (AWS, GCP, Azure, Alibaba, etc) generally maintain custom patch sets anyway.

As far as I can recall, the vast majority of modern MySQL CVEs have been low severity and/or require the attacker to already be in your private network.

For sake of comparison -- does Postgres backport every security patch to every major.minor tree? And aren't HA Postgres upgrades generally more difficult than MySQL ones since logical replication is less widespread in the PG landscape?

> For sake of comparison -- does Postgres backport every security patch to every major.minor tree?

Unless the security fix in question is not applicable to some branch, yes.

https://www.postgresql.org/support/security/ https://www.postgresql.org/support/versioning/

> Is there another level of patches below point releases for backported security patches

No, because its unecessary (in my experience)

> or are your options to upgrade to something that adds new features and may be backwards incompatible

It won't be backwards incompatible, otherwise they would leave the change for a new minor release.

Why would you not want a new feature (e.g. the new "clone plugin") if it gives you value now without any migration required (because of incompatibilities that may arise if all new features are left for minor versions which are reserved for incompatibilities)?

> No, because its unecessary (in my experience)

Assuming that is true, this sounds like a normalcy bias to me (https://en.wikipedia.org/wiki/Normalcy_bias).

> Why would you not want a new feature (e.g. the new "clone plugin")

Because features can introduce bugs or performance regressions. Sure you can say that's rare, but I still might not want to take that risk with some of the most critical infrastructure in my app.

SemVer is a relatively recent proposal that the vast majority of software does not follow. Calling the three parts of a three-part version “major”, “minor” and “patch” is a SemVer thing, not a universal standard.

SemVer did not introduce major / minor / patch, that was standard usage in free and open source software for decades beforehand.

Example from 20 years ago:

> The most popular version numbering scheme stems from the <major>.<minor>.<patchlevel> scheme

— https://ask.slashdot.org/story/01/01/05/0054230/version-numb...

Example from 1997:

> Python versions are numbered A.B.C or A.B. A is the major version number -- it is only incremented for major changes in functionality or source structure. B is the minor version number, incremented for less earth-shattering changes to a release. C is the patchlevel -- it is incremented for each new patch release.

— https://web.archive.org/web/19970501012343/http://www.python...

There's also Windows with it's A.B.C.D or major.minor.patch.build version scheme. Not sure how far back that goes, but it clearly predates semver as well.

Fair enough, you’re right that it’s more widespread than just SemVer (though IMO still not an ironclad standard)

Worth calling out here that SemVer has been around for a decade now, so "recent" is relative

For some of us, that is recent.

I'm coming to realize that semver is a lot like scrum. Everyone knows about it, lots of people like it, plenty of people want to use it, nobody seems able to move past arguing about how it's supposed to work and actually get a working implementation off the ground.

Sure, but you're assuming they follow that standard simply because they have x.y.z.

MySQL GA dates: 5.5 3 December 2010 5.6 5 February 2013 5.7 21 October 2015 8.0 19 April 2018

The average seems to be about 2.5 years between minor releases, so this doesn't seem unusual.

Usually new minor versions are only released if there are: * Changes to database behaviour defaults (e.g. default sql mode) * Deprecation or removal of features * Compatibility changes (changes to replication formats)

New features are fine, as long as they don't introduce regressions in other areas.

Even MySQL 5.7 received some good improvements in the past 18-24 months (not just bug fixes).

I feel this is a really pedantic complaint from someone who either doesn't use MySQL, or wants to hate it for a silly reason.

There are new features introduced in 8.0.26 as in .25 .24 etc before that check out release notes.

> At the scale of Facebook I wonder if they are the largest MySQL user on the planet

I have to imagine they use other databases as well, right? We're much smaller than Facebook and we have some Mongo and some Postgres and a mix of Go/Rust/Scala/node.js.

I wonder how many boxes they have dedicated to be being MySQL machines. I wonder how big their tables are and how many writes per second they see during the day.

Their scale is enormous. I first heard the phrase "thundering herds" in a Facebook MySQL talk. (Thundering herds refers to what happens to your infra when something big happens in real life that lights up social networking/search/etc.)

Let's try to put a guess in numbers.

> With roughly 2.85 billion monthly active users as of the first quarter of 2021

> 1.8 billion of Facebook users (66%) use the app on a daily basis

Say 1 database could store the data + handle the reads/writes for 100,000 users (literally just guessing)

2.85 billion monthly active users / 100k users per server = 28,500 servers/"instances" for MySQL

That's not even counting Kubernetes pods (if they use that) running Docker containers (if they use that) for their API / content serving.

I wonder what their average transactions per second looks like.

For Twitter:

> Every second, on average, around 6,000 tweets are tweeted on Twitter, which corresponds to over 350,000 tweets sent per minute, 500 million tweets per day and around 200 billion tweets per year.

> As of the first quarter of 2019, Twitter averaged 330 million monthly active users

Twitter has 11.5% the MAU of Facebook (330m of 2.85b)? I don't know if 1 tweet can be extrapolated to 1 Facebook like/comment/share/post but if it did, 6k tweets per sec * 8.7x the user base = 52.2k writes per second?

Found this:

> Facebook now has about 30,000 servers supporting its operations, hosts 80 billion photos, and serves up more than 600,000 photos to its users every second.

Kubernetes, no. Twine (aka "Don't call it Tupperware"), yes

https://engineering.fb.com/2019/06/06/data-center-engineerin...

Also, that "30,000 servers" number is off by... a lot. No comment on by how many orders of magnitude. :-)

Yep. Here's how I'd napkin math it.

Facebook has purchased 6GW of renewable energy to power its datacenters [1]. Its datacenters have a PUE of about 1.06 last they published (they unfortunately seem to have taken their dashboards down) so that's about 5.6GW of electricity actually powering real load. Some of that will power the networking fabric so we could conservatively estimate 5GW of compute load. The OCP Yosemite v3 platform power budget is a max of 1.5kW [2]. Also conservatively, let's assume a real average 1kW load. I'll leave the division as an exercise for the reader to estimate the number of servers they operate.

1 https://www.google.com/amp/s/www.euronews.com/green/amp/2021...

2 https://www.opencompute.org/documents/ocp-yosemite-v3-platfo...

5 Gigawatts to Kilowatts = 5000000

5000000 kilowatts / 1kW per machine = 5,000,000 machines internationally?

Yes, and that's a conservative lower bound (their web application servers likely draw much less than a kilowatt each and constitute a healthy portion of the fleet).

Global warming wise, should society really be focusing on using paper straws or should Facebook aim to make their server code 10% more efficient?

The good news is that electricity still costs a lot, even at scale, so there is lots of financial incentive for hyperscalars to make their datacenters and computation maximally efficient. To wit: there are a lot of people at Facebook (and Google, and Amazon, and Microsoft, and Baidu....) who work on making their server code more efficient. Small changes can result in millions of dollars a year in savings (in electricity cost, infrastructure cost, and opportunity cost in not needing to procure more servers).

We used MySQL at Dropbox too, and they probably still do.

Youtube also use MySQL.

YouTube used to use a MySQL-based system called Vitess[0]. It no longer uses it and has migrated to Spanner.

[0]: https://vitess.io/

Thanks for the update! I found more details there: https://opensource.google/projects/vitess

> How Google uses Vitess

> Vitess was serving all YouTube database traffic from 2011 to 2019.

Ah this is interesting. Do you have any references on their migration to Spanner?

gresrun works at YouTube, if by reference you mean citation then his comment is it.

wikibob was probably looking more for public documentation about the why and how rather than confirmation that the statement is true.

Yep, I love watching conference talks or reading Twitter threads, papers, etc about these kind of migrations.

Probably because Youtube was using MySQL before being acquired and switched to a better, probably fully managed DB that was much easier (free?) to maintain and scale.

was that migration a technical decision or a political one?

This is one of those scenarios where the answer is likely to be “both”, but I doubt you’ll get anyone discussing YouTube internal architecture in enough detail to be satisfying. Just look up what Vitesse is and compare what that implies in terms of operational complexity to using something like spanner if you want a straw man technical rationale.

google internally has always had a split mind about mysql. It never really jibed with the underlying google architecture but played an important role in multiple products (at least partly due to its replication features). Replacing it completely has turned out to be a huge challenge. I gained a fair amount of respect for mysql after seeing it applied at scale, but it was also a bit scary.

Wow. Why don't they use in-house Google databases, which are supposedly vastly more scalable than MySQL?

YouTube built Vitess, which lets you scale out MySQL writers. If I can scale out both my writers and my readers then what could be "vastly more scalable" than that?

Spanner paper:

https://static.googleusercontent.com/media/research.google.c...

It is designed from the ground-up to be globally distributed. Some other databases like FoundationDB (now owned and killed? by Apple) and CockroachDB were designed to scale across multiple datacenters.

Correction: FoundationDB is alive as open-source again, yay!

FoundationDB was re-released as open source a couple of years ago.

https://www.foundationdb.org/

AFAIK, FoundationDB is alive and well. What makes you think it's killed?

It was acquired by a huge company that sometimes kills products. Shortly after the acquisition, the downloads were removed from the website. It did go open source "recently" though which made it longer lasting, but I'd understand if someone would think it was dead by now.

- https://news.ycombinator.com/item?id=9259986 - Apple Acquires FoundationDB - March 24, 2015

- Comparison between March 25 and March 14, 2015 - https://web.archive.org/web/20150314231702/https://foundatio... (14th March) > https://web.archive.org/web/20150325003252/https://foundatio... (25th March)

As far as I can tell, Vitess still has trouble with cross-server transactions and distributed locking. Presumably something "vastly more scalable" could work around this.

The Spanner team noticeably put out https://cloud.google.com/blog/products/databases/inside-clou... a few years back, which tl;drs that they managed to get a reliable enough network that partitions don't really happen anymore and so you can get both C _and_ A from the CAP theorem. I don't think that Vitess is quite at that level, especially since you can self-host it and thus the devs don't have such network-level uptime guarantees.

Are cross-server transactions scalable? Would anyone architect a system that did them en masse unless they absolutely 100% had to?

I am just speculating here but I doubt that YouTube migrated to Spanner so that they could do cross-server transactions. I would think consolidating the Ops burden into the GCP org and also serving as a trophy "customer" were higher on the list of reasons. But again, just speculating...

Yeah, Google policy was to migrate everything to Spanner, not a technical decision.

> Are cross-server transactions scalable? Would anyone architect a system that did them en masse unless they absolutely 100% had to?

As a long time Vitess user and contributor/maintainer, this perfectly describes the issue. Minimal effort has been put into solving cross shard / 2PC transactions because a well designed system rarely needs it.

I guess people from PlanetScale will know. Not sure if they are reading the thread.

Because there is a substantial cost to such migrations. If it works don't fix it.

Does anything "just work"? Everything is constantly changing, new features always in demand, more users, more data.

And you need to support that for anything you have ON TOP of any migration work - which is huge, and also a similarly moving target

While "if it ain't broke, don't fix it" is often a perfectly valid approach, there are going to be times when an upgrade can result in significant performance improvements or fewer admin headaches.

If IT is on the value chain this seems insane. If IT is ancillary to the value chain then this is standard.

I don't think the people behind Spanner are a bunch of 20yo throwing together a MongoDB

GitHub also seems to use MySql, as recently as 2020[1]

[1] https://github.blog/2020-05-20-three-bugs-in-the-go-mysql-dr...

Still do

The NSA also uses MySQL for their project XKEYSTORE (digital surveillance)

Any idea what Facebooks stores in MySQL? Core user database?

To a first approximation, everything.

Most product engineers interact with it by querying TAO, which you can read about in various blog posts, but is basically a cache in front of MySQL (this is a vast oversimplification).

Hi, Kevin. :) As of the time I left, the bulk storage system I worked on - images, video, machine-learning datasets, backups of other things, etc. - held most of the bytes at Facebook. The MySQL/TAO complex had orders of magnitude more addressable objects, undoubtedly handled many times more requests and bytes in flight per second, but when it comes to bytes at rest they're not even number one.

Fair enough :)

Basically all the small stuff. Every user's identity, preferences, history, and connections. If you post text, that's there too. If you post media, that's somewhere else. Connections are the hard part, because it's a gigantic graph with the potential to make every query - e.g. to construct a user's timeline - hit vast numbers of servers. There's all sorts of special sauce to do with partitioning, caching, etc. to absorb as much as possible of that load before it hits the MySQL servers, but it's still a gigantic service even by Facebook standards.