News for hackers. Stuff that matters.
Re:Scaling Bitbucket’s Database by adamfeldman on Saturday February 15, 2020 @02:25PM (#22337323)
"Bitbucket uses PostgreSQL with one primary read-write database and N read-only replicas."
Re:Scaling Bitbucket’s Database by hinkley on Sunday February 16, 2020 @02:11AM (#22340280)
I am architecting a project that is at a smaller scale but bears some resemblance, and I’ve already settled that I will be very careful about GET vs POST/PUT semantics and route all traffic of the former only to the replicas and all of the latter through a connection pool to the primary. That, I expect, will hold me well into >1% of their traffic.

I’m not sure if I’ve missed something with their LSN work or if this indicates that the GET semantics horse has already escaped their proverbial barn. Within the call-response of a single request, which they seem to be talking about, none of this should be necessary. Right?

However a cascade of narrowly-spaced follow-up requests could easily catch you in this trick. Lie. Whatever you wish to call it.

Re:Scaling Bitbucket’s Database by cryptonector on Sunday February 16, 2020 @01:03AM (#22340053)
Bitbucket is a perfect case for sharding. You could have one DB of users, and N DBs of user data, with the first referencing the latter, and then distribute user data among the latter. You could do this straight up with PostgreSQL without any special server-side software. You could also have a PG server using FDW to act as a proxy for all the DBs a client needs to be talking to. There are many other options too.
Re:Scaling Bitbucket’s Database by zzzeek on Saturday February 15, 2020 @02:39PM (#22337429)
Bitbucket (mostly via Atalassian) worked very hard to chase everyone off their platform in the vain hopes that the OSS community would start using Jira. Clearly they bet wrong, but unfortunately they aren't going to get any users back for a very long time.
Re:Scaling Bitbucket’s Database by dreamcompiler on Saturday February 15, 2020 @07:14PM (#22338956)
I've already moved 100 repos from BB over to Github. Free private repos on Github were not enough to get me to go to the trouble of moving, but then BB deprecated Mercurial, so there's no longer any reason to stay. I'm gone.
Re:Scaling Bitbucket’s Database by kyrra on Saturday February 15, 2020 @07:27PM (#22339020)
I just went through this as well. And I discovered that GitHub can import private bitbucket mercurial projects. That made it super easy.
Re:Scaling Bitbucket’s Database by dreamcompiler on Saturday February 15, 2020 @08:53PM (#22339344)
Agreed. Turned out to be much easier than I expected.
Re:Scaling Bitbucket’s Database by quadrifoliate on Saturday February 15, 2020 @05:32PM (#22338495)
I also don't see anything groundbreaking about Bitbucket as a Git repo interface. GitHub is miles ahead in both polish and mindshare.

Free private repos used to be the one thing keeping me (and from conversations, several people I know) on Bitbucket and once GitHub offered those as well, there was no incentive to stay. The Mercurial support was probably another unique feature they had from way back but they seem to have decided to remove that too.

Re:Scaling Bitbucket’s Database by zzzeek on Sunday February 16, 2020 @09:09AM (#22341601)
yes, at the beginning, we thought github and bitbucket would be in a healthy competition for features and polish. But then Atlassian bought Bitbucket, and their agenda was, "make BB as awful as possible so we can steal whoever into our crappy corporate software" and that was that. I had to move not just because the issue tracker was garbage but also because Bitbucket's UX became horrible, not just the slowness but also the layout was very broken, having to read diffs inside of a tiny scrolling pane and things like that.
Re:Scaling Bitbucket’s Database by freedomben on Saturday February 15, 2020 @06:45PM (#22338834)
Yep, it was private repos for me. Once Gitlab came about and offered free private repos I bailed as quickly as I could from Bitbucket.
Re:Scaling Bitbucket’s Database by alexis_fr on Saturday February 15, 2020 @03:24PM (#22337722)
Free private repos is a real upside.

Central login where you can never understand what login you are using and what information they are collecting, nor on which website you are not whether you’ve suddenly integrated Confluence into a Bitbucket instance, is a real downside. God I hate SSO’ed companies, they went all the way of Google into Youtube.

Re:Scaling Bitbucket’s Database by progval on Saturday February 15, 2020 @05:20PM (#22338426)
> Free private repos is a real upside.

which isn't true anymore, as other pointed out.

Also offering Mercurial was a reason for choosing Bitbucket, but they are ending its support.

Re:Scaling Bitbucket’s Database by detaro on Saturday February 15, 2020 @04:19PM (#22338054)
Free private repos are an upside over ... Github, which gives free accounts unlimited private repos? Gitlab, which does the same?
Re:Scaling Bitbucket’s Database by Cyph0n on Saturday February 15, 2020 @04:56PM (#22338278)
To be fair, Github only added that recently.
Re:Scaling Bitbucket’s Database by zzzeek on Saturday February 15, 2020 @04:17PM (#22338047)
github offers these now for individual accounts.
Re:Scaling Bitbucket’s Database by agumonkey on Saturday February 15, 2020 @04:57PM (#22338292)
this reminds me that I used to use bitbucket for the private repo and I just hosted some app code on .. github private.
Re:Scaling Bitbucket’s Database by emmelaich on Saturday February 15, 2020 @04:12PM (#22338014)
Do you have a link or two for that?
Re:Scaling Bitbucket’s Database by zzzeek on Saturday February 15, 2020 @04:16PM (#22338043)
nope just some email exchanges with employees which started with begging them for the most rudimental Bitbucket issue tracker features and fixes and ended with semi-explicit admissions that their hands were tied due to the "just move to Jira" corporate directive.

See "Want a more powerful issue tracker?" at the bottom of https://confluence.atlassian.com/bitbucket/issue-trackers-22... for the documentation of this policy.

Re:Scaling Bitbucket’s Database by caseyf7 on Sunday February 16, 2020 @12:46AM (#22340005)
Does storing the LSNs in Redis and Elasticache create the same race condition they are trying to avoid or am I missing something?
Re:Scaling Bitbucket’s Database by nihil75 on Saturday February 15, 2020 @02:33PM (#22337386)
Yea I'm not taking advice from a service that's down once a week
Re:Scaling Bitbucket’s Database by izacus on Saturday February 15, 2020 @02:37PM (#22337420)
Did you actually attempt to read the article or did you just come here to spew out needless negativity?
Re:Scaling Bitbucket’s Database by chairmanwow1 on Saturday February 15, 2020 @03:12PM (#22337649)
Have you ever actually used Bitbucket in production? It has absurd latencies for every operation from website to even a git push taking full seconds to respond.

It’s hard to take them seriously when that’s the tableau upon which they are advising us to build systems.

Re:Scaling Bitbucket’s Database by skunkworker on Saturday February 15, 2020 @05:47PM (#22338568)
A number of years ago I had all of my repos on bitbucket, but recently I’ve pulled everything as their peering can impact large git pulls and were unreliable. The transition with the account changeover was enough to make me jump ship to github completely.
Re:Scaling Bitbucket’s Database by nihil75 on Sunday February 16, 2020 @12:32PM (#22342646)
This.
Re:Scaling Bitbucket’s Database by YawningAngel on Saturday February 15, 2020 @07:58PM (#22339144)
I don't think it's needless. I think the fact that Bitbucket is profoundly second-rate in is quite relevant when discussing their fluffy blog posts about engineering.
Re:Scaling Bitbucket’s Database by pm90 on Sunday February 16, 2020 @01:28AM (#22340144)
I’m not a big fan of Atlassian in general but the blog post provides a reasonable solution to an engineering problem. It doesn’t seem like a fluff piece.
Re:Scaling Bitbucket’s Database by rosstafarian0 on Saturday February 15, 2020 @06:16PM (#22338722)
Is it just me or does looking at write logs to determine read consistency just seem like a really bad way to do this vs a different load balancing strategy?
Re:Scaling Bitbucket’s Database by hilbertseries on Saturday February 15, 2020 @07:27PM (#22339015)
Definitely feels like bandaid, over a more robust solution like sharding their main database by repository. Or splitting off some of their tables to another database. Their load of "millions of requests per hour", doesn't sound particularly high either.

At some point their primary is going to get overloaded again by the writes. And they'll have added all of this machinery for nothing. They've also made the replicas an essential part of their stack. Whereas their previous stack could probably tolerate a certain amount of replicas going down or replica lag. They now have a system that will grow and be dependent on all the replicas being available. Until a heavy write load or heavy table alter, causes the replicas to become lagged at which point a higher percentage of traffic will go to the master and potentially cause downtime.

Re:Scaling Bitbucket’s Database by pm90 on Sunday February 16, 2020 @01:33AM (#22340154)
Yikes. Excellent point. Is sharding the only option here? It seems like having immensely shared Postgres is one way of solving the problem, but is that the only option?

Wondering if eg cockroachDB would be a good fit

Re:Scaling Bitbucket’s Database by hodgesrm on Saturday February 15, 2020 @11:53PM (#22339857)
This is a very common solution. The biggest advantage is that it allows you to use read replicas which are there for HA and otherwise idle.
Re:Scaling Bitbucket’s Database by ykevinator on Sunday February 16, 2020 @09:06AM (#22341582)
It's amazing that Atlassian still exists.
Re:Scaling Bitbucket’s Database by lykr0n on Saturday February 15, 2020 @02:47PM (#22337488)
Curious that they use LSN tracking vs making the replication synchronous. I would be curious to see the performance numbers and the reasoning behind the choice.
Re:Scaling Bitbucket’s Database by beoberha on Saturday February 15, 2020 @03:08PM (#22337629)
Synchronous replication really slows down write performance and adds in the complexity of needing to determine if a replica is down. I think very few people use it in practice.
Re:Scaling Bitbucket’s Database by Xorlev on Saturday February 15, 2020 @03:57PM (#22337916)
+1, sync replication also reduces availability: any replica downtime becomes write unavailability until you decide that replica is "dead" and stop replicating to it.

There's also hybrid setups of sync+async for durability reasons, sync to a single replica so that if your primary db suddenly catches fire you have a replica that's as up to date as possible, then read replicas using async replication.

Re:Scaling Bitbucket’s Database by saga92 on Saturday February 15, 2020 @06:45PM (#22338835)
Oracle offers DML redirection for active dataguard setup where a write operation on the replica database is redirected to the primary database to allow applications that make infrequent writes to actively run on the Active Data Guard replica database. Also the write operation completes when the replica has seen the write from the primary thereby eliminating the race condition avoiding such complex reengineering
Re:Scaling Bitbucket’s Database by rosstafarian0 on Saturday February 15, 2020 @06:48PM (#22338850)
A lot of databases have a concept of consistent reads. Even DynamoDB will allow you to specify consistent reads explicitly on a query and writing through DAX will take care of the scenario as you described as the read cache is immediately updated. This a common scenario.
Re:Scaling Bitbucket’s Database by loic-sharma on Saturday February 15, 2020 @02:26PM (#22337334)
That's really neat! I wonder, could you store the PostgreSQL log sequence numbers in a cookie to save the round-trip to Redis?
Re:Scaling Bitbucket’s Database by nhoughto on Saturday February 15, 2020 @06:18PM (#22338729)
I thought this too, but that then creates a likely unreasonable burden on the client. Not all user agents expect cookies either, think of all the varieties REST clients. Sounds like they would also need to internally reference the lsn too, so being able to look it up internally would be valuable.

Feels like there is something there tho in terms of approach, esp since they are keying it on user id.

Re:Scaling Bitbucket’s Database by halestock on Saturday February 15, 2020 @07:12PM (#22338946)
I found it interesting that their cloud version of bitbucket uses Django while their self-hosted stack is almost entirely Java.
Re:Scaling Bitbucket’s Database by mithun on Saturday February 15, 2020 @07:22PM (#22338994)
That's because they bought Bitbucket. The self hosted version was internally developed and used to be called Stash, before they rebranded.
Re:Scaling Bitbucket’s Database by sedatk on Sunday February 16, 2020 @12:56AM (#22340031)
Step 1: Remove Mercurial support.
Re:Scaling Bitbucket’s Database by ngrilly on Saturday February 15, 2020 @05:49PM (#22338575)
> we find a replica that is as up to date as the user's saved LSN

The article doesn't explain how they know the current LSN of each replica?

Re:Scaling Bitbucket’s Database by avremel on Saturday February 15, 2020 @09:03PM (#22339378)
IIUC they need to query replicas until they find one that is up to date.

"We have to query Elasticache and replicas, which adds latency."

Re:Scaling Bitbucket’s Database by qeternity on Saturday February 15, 2020 @03:10PM (#22337642)
10ms to do a redis get? What am I missing here...
Re:Scaling Bitbucket’s Database by Xorlev on Saturday February 15, 2020 @03:55PM (#22337905)
They said that was their time budget, but didn't follow up with how expensive it ended up being in practice -- but agreed.
Re:Scaling Bitbucket’s Database by qeternity on Sunday February 16, 2020 @06:41PM (#22344443)
The article says they were able to keep it close to their max budget, which actually suggests that it was higher than 10ms in practice
Re:Scaling Bitbucket’s Database by avremel on Saturday February 15, 2020 @09:05PM (#22339384)
And presumably query the replicas until an up-to-date one is found (based off the LSN of the user).
Re:Scaling Bitbucket’s Database by qeternity on Sunday February 16, 2020 @06:40PM (#22344436)
I’d have expected the LSN of the replicas to be in redis as well, but fair point. However even as a worst case given their number of replicas, I’d expect to be able to hit redis AND each replica in under 10ms
Re:Scaling Bitbucket’s Database by doubleunplussed on Saturday February 15, 2020 @02:18PM (#22337278)
Still salty about them dropping mercurial support without providing any migration tools, and planning to just delete data without any archiving.
Re:Scaling Bitbucket’s Database by compumike on Saturday February 15, 2020 @05:19PM (#22338417)
Also salty!

For hg->git migration, https://github.com/frej/fast-export worked smoothly for me.

    mkdir new-git-repo
    cd new-git-repo
    git init
    fast-export/hg-fast-export.sh -r ../path-to-old-hg-repo
    (probably want to translate .hgignore to .gitignore and commit it)
    git remote add origin git@path_to:git_server
    git push -u origin master
    git push --all
Re:Scaling Bitbucket’s Database by johannes1234321 on Saturday February 15, 2020 @07:54PM (#22339128)
Yes, it is fairly easy to convert the repo, but

1) this afaik couldn't be done inside the BB project, so one would have had to create a new repo and lose bug history etc.

2) It's a trust thing. If BB kills repos once, who doesn't say they won't kill others tomorrow?

3) Comfort: Import to GitHub is a click away

Re:Scaling Bitbucket’s Database by michaelhoffman on Saturday February 15, 2020 @05:46PM (#22338561)
Same here. I don't love that they aren't going to support Mercurial anymore. But deleting all our existing projects without providing a path for us to migrate or archive? I can't trust them anymore.
Re:Scaling Bitbucket’s Database by meritt on Saturday February 15, 2020 @02:36PM (#22337416)
I'm also salty. What's your plan? I'm either gonna migrate to git (and use a different host), or self-host something like hetapod.net
Re:Scaling Bitbucket’s Database by bjoli on Saturday February 15, 2020 @03:26PM (#22337739)
I went to sourcehut. I am very happy, but during the process I ended up actually having to learn a bit of git. Future projects will probably be git.
Re:Scaling Bitbucket’s Database by doubleunplussed on Sunday February 16, 2020 @02:07AM (#22340264)
Yeah I'm moving to git and github. I always pteferred mercurial for what I maintain are good reasons not just of personal taste, but I've been swimming against the tide for too long and it makes sense to go with git and github.

If github goes downhill, at least I'll be in good company next time!

Re:Scaling Bitbucket’s Database by patd on Saturday February 15, 2020 @04:35PM (#22338152)
GitHub’s import feature worked well for me. You’re obviously going to lose the wiki, issues etc. But at least you’ll keep your repo history, branches, ...