I've used Coinbase on and off for a while. The service always felt exactly as you describe it: startup busting their collective ass to keep up with demand.

Given the recent increase in news about cryptocurrency I've expected "technical difficulties" with Coinbase. Heck, most mining pools are sweating bullets right now.

Hypergrowth = hyperpains.

missed opportunity to say:

hypergains = hyperpains

no hyperpains no hypergains

hyperpains are just hyperweaknesses leaving your body.

I would guess they're also likely the first stop for new entrants, as they're (or were?) the only US based exchange, easiest to use, and insured by FDIC/Lloyd's.

Coinbase has been my recommendation to anyone new to the scene who wants to buy/use bitcoin.

And obviously once finished, move all funds to paper wallet.

Which paper wallet software do you suggest?

Is it really that much traffic compared to your typical consumer-facing start up? There can't be that many people buying BTC compared to the number of people that say, Stripe or Whatsapp handled in their early days...

I don't know about Stripe or Whatsapp, but I know coinbase has had an average of 30,000 new users a day for a while, with spikes of over 100,000 new signups in a single day around thanksgiving.

It's absolutely bonkers amounts of growth!

Edit: more concrete numbers:

In January of 2016 they had around 2.6 million users.

On November 29th they announced they had over 13 million users.

Whatsapp famously had ~450 million users when they were acquired by Facebook. I assume that the average user sent quite a few texts, some pictures, occasional video... they also supported voice calls...

I don't know how much computing power is actually needed when you're just buying BTC. When I used Coinbase recently to take a small BTC position, I only made maybe 4 transactions in total. Is this really a lot to handle?

I don't have any experience in this area, but I imagine that fintech is a bit different than messaging. It's not the amount of data, but the nature of the data.

I'd imagine you can't use scaling tricks like eventual consistency or simple sharding here. Traders are going to be pissed if their most up to date information from the exchange is a second behind what they can see.

It's already at the point where they have a line in their FAQ for the GDAX API that tells people where to colocate from for the lowest latency.

And all of that data needs to be stored damn-near indefinitely, and securely. And integrated with traditional financial systems.

I'd love to see some information on what it actually takes, as i'm just guessing here, but seeing as there aren't any bitcoin exchanges that can seem to keep the servers running smoothly during spikes, i'm guessing it's not a trivial problem to solve.

Thanks, I'm not in finance but it seems obvious in retrospect that not being able to rely on things being eventually consistent is a bottle neck. Before thinking about this, the number of users that people threw around when talking about BTC exchanges seemed very small.

And yet the NYSE keeps on ticking.

... from 9 to 5, monday through friday, with a fairly predictable load, and decades more experience.

I'm not saying it's impossible, but the amount of growth they have seen is substantial and unprecedented. Nobody could have predicted it, and the rate of increase just keeps going up.

They are clearly new in this space, as all cryptocurrency companies are. I don't think that means that they should just give up and go home because they couldn't keep the servers up during spikes of insane activity...

> ... from 9 to 5, monday through friday, with a fairly predictable load, and decades more experience.

And a lot more operating capital, too.

There’s a lot less dangers involved in double-delivering Whatsapp messages, delivering them out of order, etc. Less constraints=easier optimizations.

That, followed by the fact that many of their users probably sit on the site a good part of the day constantly refreshing the page/taking up websocket bandwidth to watch price changes. Yeah, load is a serious issue.

I'm genuinely amazed that they can do streaming updated to GDAX as fast as they do.

Jan 2016 to now is 23 months. Still big but half as much.

Coinbase probably deals with lots of bot trading. Most consumer-facing startups deal with human-scale behavior, not bot-scale.

Ah that's good point, they probably generate a lot of traffic and could increase rapidly while the number of users remains somewhat small.

I think its also a matter of funding and company financials as well. You have to make money to justify spending money and each company has a different idea on what an acceptable infrastructure spend is. While it might be great to spend a ton of money and build a bullet proof infrastructure, if traffic slows down or revenue dips they are in the very negative position of having a lot of hardware and not a lot of money in the bank.

They have to scale at the rate their revenue or financial backers will be able to sustain. Based on the number of new users signing up to the service the down time is not yet a significant deterrent. If use drops because of downtime they will likely need to reevaluate a large capital spend to solve the issue.

"We want to run an exchange that the entire world will use for this crypto-currency that's become very popular and has increased in price 1000% since it became widely known."

I think I would have predicted that they'd need to run at a very large scale, and be able to handle massive spikes. Whether or not I could have convinced them to make the investment is another story.

The problem is what happens if that growth stops? What happens if bitcoin tanks right after you hired 300 new customer service reps and tripled your server hardware?

It's a gamble no matter what, but moving too quickly or moving too slowly results in the company going under.

In that case, I think straddling the line of "overwhelmed when there is a big spike" is a good spot to be in all things consitered.

You have layoffs and turn servers off. This isn't rocket surgery. They're skimming a ton of money off these trades every day, down time is a loss.

The optics on "servers are down due to a massive spike in volume" are much better than "Coinbase had another round of layoffs today..."

Currently they are signing people up at a dramatic rate despite their server issues. If they start laying people off because they spun up to fast they could start a panic that would kill them.

They have to weigh all of this in when deciding on an infrastructure spend.

I would also like to think from a purely personal level that they don't want to hire and fire people like that as these are real people with real families involved.

Coinbase goes through this charade every time this happens with a major digital currency. It wasn't a legitimate excuse the first time, and it's not an excuse the 7th time.

Scaling is a solved problem. You have AWS, Google Cloud, and MS Azure to choose from with financial institution/HIPAA-level offerings.

Bitcoin doesn't scale with "The Cloud". If anyone can get the private key, they have the money... sending it off to any other provider is going to end badly. Sure any other company doesn't care if "The Cloud" leaks.. it's just the customers problem, cause it's their data... However Bitcoin leaks, it's the companies money, so it's now again their problem.

Spoken like someone who truly knows nothing about scaling.

Imgur was reporting being over capacity just this morning.

By most engineers in the field it is a difficult problem with ongoing work.

Lol scaling is not a solved problem. Garbage collection is a solved problem. Scaling is always a challenge.

Scaling is by design, seems it was not designed to scale (serverless architecture and db)

Alas, those silver bullets don’t work either. Scaling still remains a multi-faceted, complex problem with a lot of variables.

The thing about scaling is that you can't justify the engineering involved in overscaling until you're reasonably sure you'll need the capacity. Otherwise that's called overengineering. Managers get fired for approving spending on things like that when the demand does not appear. It's also hard to simulate real traffic because they don't always behave how you predicted they would.

Also, your system gets more and more complex as you scale. More monitoring systems needed, more attack vectors to watch for, deployment systems, backup systems... Imagine the security level coinbase needs to have. It's somewhat unprecedented since a hack could make off with a lot of valuable assets in an instant. (even just from their hot-wallet)

You saying scaling is solved makes me question what you think their infrastructure looks like. It's not just an AWS loadbalancer with some shards behind it. It's A LOT more complex than that.

Slapdash startup-ism in an exchange for a Brave New Currency is just as scary as a conspiracy. It may be more scary.

At least with the Cabal theories you know someone knows what's going on. You might even have someone to blame/jail at the end of the tunnel!

You really seem to be wringing your hands pretty hard about an overloaded NGINX instance. Wouldn't a conspiracy that was stealing everyone's money be a lot worse?

No I'm saying that any reply approximating "its fine guys, they're new to this and won't anticipate everything" is disquieting when lots of coin is on the line.

> They are still scaling their infrastructure and when the price starts moving quickly everyone logs in to either get in on the action or take a look.

Are they not using cloud, which should enable easy scaling? Or is their architecture not designed for good scaling?

No architecture scales infinitely, you merely shift at each step what the next bottleneck will be. Even with small scale sites you'll see this: first, you need to move from everything on one server to a load balancer plus app servers talking to a single database. Then you need to share your database. Then you need to move to multiple load balancers. Then so on and so forth, depending on the specifics of your service, mixed in with various rounds of local optimization at the application level that buy you just a little more time before having to ship another architectural change.

Most distributed systems don't really scale ass you add nodes beyond a factor of 10 to 100. If they did, you wouldn't need to hire super expensive people to handle those systems.

The cloud enables easier scaling, it's not a silver bullet. No matter what, scaling plans have a region where they break down, so it doesn't imply they aren't designed for good scaling.

You're not a programmer, right?

Cloud solves everything guys!!! Yes buy more Cloud now that's 10% down in the exchanges lol...