Re:Scaling Bitbucket’s Database
by hinkley on Sunday February 16, 2020 @02:11AM
(#22340280)
I am architecting a project that is at a smaller scale but bears some resemblance, and I’ve already settled that I will be very careful about GET vs POST/PUT semantics and route all traffic of the former only to the replicas and all of the latter through a connection pool to the primary. That, I expect, will hold me well into >1% of their traffic.
I’m not sure if I’ve missed something with their LSN work or if this indicates that the GET semantics horse has already escaped their proverbial barn. Within the call-response of a single request, which they seem to be talking about, none of this should be necessary. Right?
However a cascade of narrowly-spaced follow-up requests could easily catch you in this trick. Lie. Whatever you wish to call it.
Re:Scaling Bitbucket’s Database
by cryptonector on Sunday February 16, 2020 @01:03AM
(#22340053)
Bitbucket is a perfect case for sharding. You could have one DB of users, and N DBs of user data, with the first referencing the latter, and then distribute user data among the latter. You could do this straight up with PostgreSQL without any special server-side software. You could also have a PG server using FDW to act as a proxy for all the DBs a client needs to be talking to. There are many other options too.
Re:Scaling Bitbucket’s Database
by zzzeek on Saturday February 15, 2020 @02:39PM
(#22337429)
Bitbucket (mostly via Atalassian) worked very hard to chase everyone off their platform in the vain hopes that the OSS community would start using Jira. Clearly they bet wrong, but unfortunately they aren't going to get any users back for a very long time.
Re:Scaling Bitbucket’s Database
by dreamcompiler on Saturday February 15, 2020 @07:14PM
(#22338956)
I've already moved 100 repos from BB over to Github. Free private repos on Github were not enough to get me to go to the trouble of moving, but then BB deprecated Mercurial, so there's no longer any reason to stay. I'm gone.
Re:Scaling Bitbucket’s Database
by quadrifoliate on Saturday February 15, 2020 @05:32PM
(#22338495)
I also don't see anything groundbreaking about Bitbucket as a Git repo interface. GitHub is miles ahead in both polish and mindshare.
Free private repos used to be the one thing keeping me (and from conversations, several people I know) on Bitbucket and once GitHub offered those as well, there was no incentive to stay. The Mercurial support was probably another unique feature they had from way back but they seem to have decided to remove that too.
Re:Scaling Bitbucket’s Database
by zzzeek on Sunday February 16, 2020 @09:09AM
(#22341601)
yes, at the beginning, we thought github and bitbucket would be in a healthy competition for features and polish. But then Atlassian bought Bitbucket, and their agenda was, "make BB as awful as possible so we can steal whoever into our crappy corporate software" and that was that. I had to move not just because the issue tracker was garbage but also because Bitbucket's UX became horrible, not just the slowness but also the layout was very broken, having to read diffs inside of a tiny scrolling pane and things like that.
Re:Scaling Bitbucket’s Database
by alexis_fr on Saturday February 15, 2020 @03:24PM
(#22337722)
Free private repos is a real upside.
Central login where you can never understand what login you are using and what information they are collecting, nor on which website you are not whether you’ve suddenly integrated Confluence into a Bitbucket instance, is a real downside. God I hate SSO’ed companies, they went all the way of Google into Youtube.
Re:Scaling Bitbucket’s Database
by zzzeek on Saturday February 15, 2020 @04:16PM
(#22338043)
nope just some email exchanges with employees which started with begging them for the most rudimental Bitbucket issue tracker features and fixes and ended with semi-explicit admissions that their hands were tied due to the "just move to Jira" corporate directive.
Re:Scaling Bitbucket’s Database
by chairmanwow1 on Saturday February 15, 2020 @03:12PM
(#22337649)
Have you ever actually used Bitbucket in production? It has absurd latencies for every operation from website to even a git push taking full seconds to respond.
It’s hard to take them seriously when that’s the tableau upon which they are advising us to build systems.
Re:Scaling Bitbucket’s Database
by skunkworker on Saturday February 15, 2020 @05:47PM
(#22338568)
A number of years ago I had all of my repos on bitbucket, but recently I’ve pulled everything as their peering can impact large git pulls and were unreliable. The transition with the account changeover was enough to make me jump ship to github completely.
Re:Scaling Bitbucket’s Database
by YawningAngel on Saturday February 15, 2020 @07:58PM
(#22339144)
I don't think it's needless. I think the fact that Bitbucket is profoundly second-rate in is quite relevant when discussing their fluffy blog posts about engineering.
Re:Scaling Bitbucket’s Database
by pm90 on Sunday February 16, 2020 @01:28AM
(#22340144)
I’m not a big fan of Atlassian in general but the blog post provides a reasonable solution to an engineering problem. It doesn’t seem like a fluff piece.
Re:Scaling Bitbucket’s Database
by rosstafarian0 on Saturday February 15, 2020 @06:16PM
(#22338722)
Is it just me or does looking at write logs to determine read consistency just seem like a really bad way to do this vs a different load balancing strategy?
Re:Scaling Bitbucket’s Database
by hilbertseries on Saturday February 15, 2020 @07:27PM
(#22339015)
Definitely feels like bandaid, over a more robust solution like sharding their main database by repository. Or splitting off some of their tables to another database. Their load of "millions of requests per hour", doesn't sound particularly high either.
At some point their primary is going to get overloaded again by the writes. And they'll have added all of this machinery for nothing. They've also made the replicas an essential part of their stack. Whereas their previous stack could probably tolerate a certain amount of replicas going down or replica lag. They now have a system that will grow and be dependent on all the replicas being available. Until a heavy write load or heavy table alter, causes the replicas to become lagged at which point a higher percentage of traffic will go to the master and potentially cause downtime.
Re:Scaling Bitbucket’s Database
by pm90 on Sunday February 16, 2020 @01:33AM
(#22340154)
Yikes. Excellent point. Is sharding the only option here? It seems like having immensely shared Postgres is one way of solving the problem, but is that the only option?
Re:Scaling Bitbucket’s Database
by lykr0n on Saturday February 15, 2020 @02:47PM
(#22337488)
Curious that they use LSN tracking vs making the replication synchronous. I would be curious to see the performance numbers and the reasoning behind the choice.
Re:Scaling Bitbucket’s Database
by beoberha on Saturday February 15, 2020 @03:08PM
(#22337629)
Synchronous replication really slows down write performance and adds in the complexity of needing to determine if a replica is down. I think very few people use it in practice.
Re:Scaling Bitbucket’s Database
by Xorlev on Saturday February 15, 2020 @03:57PM
(#22337916)
+1, sync replication also reduces availability: any replica downtime becomes write unavailability until you decide that replica is "dead" and stop replicating to it.
There's also hybrid setups of sync+async for durability reasons, sync to a single replica so that if your primary db suddenly catches fire you have a replica that's as up to date as possible, then read replicas using async replication.
Re:Scaling Bitbucket’s Database
by saga92 on Saturday February 15, 2020 @06:45PM
(#22338835)
Oracle offers DML redirection for active dataguard setup where a write operation on the replica database is redirected to the primary database to allow applications that make infrequent writes to actively run on the Active Data Guard replica database. Also the write operation completes when the replica has seen the write from the primary thereby eliminating the race condition avoiding such complex reengineering
Re:Scaling Bitbucket’s Database
by rosstafarian0 on Saturday February 15, 2020 @06:48PM
(#22338850)
A lot of databases have a concept of consistent reads. Even DynamoDB will allow you to specify consistent reads explicitly on a query and writing through DAX will take care of the scenario as you described as the read cache is immediately updated. This a common scenario.
Re:Scaling Bitbucket’s Database
by nhoughto on Saturday February 15, 2020 @06:18PM
(#22338729)
I thought this too, but that then creates a likely unreasonable burden on the client. Not all user agents expect cookies either, think of all the varieties REST clients. Sounds like they would also need to internally reference the lsn too, so being able to look it up internally would be valuable.
Feels like there is something there tho in terms of approach, esp since they are keying it on user id.
Re:Scaling Bitbucket’s Database
by qeternity on Sunday February 16, 2020 @06:40PM
(#22344436)
I’d have expected the LSN of the replicas to be in redis as well, but fair point. However even as a worst case given their number of replicas, I’d expect to be able to hit redis AND each replica in under 10ms
Re:Scaling Bitbucket’s Database
by michaelhoffman on Saturday February 15, 2020 @05:46PM
(#22338561)
Same here. I don't love that they aren't going to support Mercurial anymore. But deleting all our existing projects without providing a path for us to migrate or archive? I can't trust them anymore.
Re:Scaling Bitbucket’s Database
by doubleunplussed on Sunday February 16, 2020 @02:07AM
(#22340264)
Yeah I'm moving to git and github. I always pteferred mercurial for what I maintain are good reasons not just of personal taste, but I've been swimming against the tide for too long and it makes sense to go with git and github.
If github goes downhill, at least I'll be in good company next time!
Re:Scaling Bitbucket’s Database
by patd on Saturday February 15, 2020 @04:35PM
(#22338152)
GitHub’s import feature worked well for me. You’re obviously going to lose the wiki, issues etc. But at least you’ll keep your repo history, branches, ...
I’m not sure if I’ve missed something with their LSN work or if this indicates that the GET semantics horse has already escaped their proverbial barn. Within the call-response of a single request, which they seem to be talking about, none of this should be necessary. Right?
However a cascade of narrowly-spaced follow-up requests could easily catch you in this trick. Lie. Whatever you wish to call it.
Free private repos used to be the one thing keeping me (and from conversations, several people I know) on Bitbucket and once GitHub offered those as well, there was no incentive to stay. The Mercurial support was probably another unique feature they had from way back but they seem to have decided to remove that too.
Central login where you can never understand what login you are using and what information they are collecting, nor on which website you are not whether you’ve suddenly integrated Confluence into a Bitbucket instance, is a real downside. God I hate SSO’ed companies, they went all the way of Google into Youtube.
which isn't true anymore, as other pointed out.
Also offering Mercurial was a reason for choosing Bitbucket, but they are ending its support.
See "Want a more powerful issue tracker?" at the bottom of https://confluence.atlassian.com/bitbucket/issue-trackers-22... for the documentation of this policy.
It’s hard to take them seriously when that’s the tableau upon which they are advising us to build systems.
At some point their primary is going to get overloaded again by the writes. And they'll have added all of this machinery for nothing. They've also made the replicas an essential part of their stack. Whereas their previous stack could probably tolerate a certain amount of replicas going down or replica lag. They now have a system that will grow and be dependent on all the replicas being available. Until a heavy write load or heavy table alter, causes the replicas to become lagged at which point a higher percentage of traffic will go to the master and potentially cause downtime.
Wondering if eg cockroachDB would be a good fit
There's also hybrid setups of sync+async for durability reasons, sync to a single replica so that if your primary db suddenly catches fire you have a replica that's as up to date as possible, then read replicas using async replication.
Feels like there is something there tho in terms of approach, esp since they are keying it on user id.
The article doesn't explain how they know the current LSN of each replica?
"We have to query Elasticache and replicas, which adds latency."
For hg->git migration, https://github.com/frej/fast-export worked smoothly for me.
1) this afaik couldn't be done inside the BB project, so one would have had to create a new repo and lose bug history etc.
2) It's a trust thing. If BB kills repos once, who doesn't say they won't kill others tomorrow?
3) Comfort: Import to GitHub is a click away
If github goes downhill, at least I'll be in good company next time!