News for hackers. Stuff that matters.
Re:Could Better Testing Practices Have Prevented the Healthc... by programminggeek on Tuesday October 08, 2013 @12:30PM (#6516035)
I've dealt with having to scale a site to millions of pageviews and honestly, if you've never had to deal with a traffic spike, you probably aren't going to build for the scale issues you will have.

For example, there is a lot of caching you just have to do. Tons of it. Cache as much as you can. Memcache and Varnish are your friends, use them as much as you can. Unfortunately, if Healthcare.gov was in a very write-heavy situation, they are somewhat limited in how much caching will help.

One thing they could have done that would have saved A TON of load is not require users to sign up before giving them a list of available plans. That whole part didn't need to be database driven at all. The data could have been stored in redis and they could have used javascript to filter it based on a form. That would have been ridiculously fast. They also could have prerendered all the plan list possibilities and stored those in varnish. That also would have been ridiculously fast. My guess is millions of people just wanted to check prices and eliminating the database load for those users would have probably kept things running fast and smooth.

Slow DB queries are the enemy and you don't realize how bad they are until you are at scale. Sure, it only takes a few seconds on your local machine, but multiply that times thousands of concurrent users and your DB gets swamped. If you are using an ORM, it is MUCH harder to track down where in your code that 3 way join that scans every record is happening. Ideally you'd be using straight SQL and maybe use comments to tag a query. Also tools like newrelic might help if only because many databases don't offer great visibility of performance data.

Sharding your database is something that is probably possible, and in the case of healthcare.gov, they probably could have had totally separate infrastructure on a per-state basis that would have made scaling a lot easier than putting everything on the same database. Also, put reporting and things that aren't mission critical on a slave database. The last thing you want is a reporting job bringing down the live site in the background.

Getting good hardware with fast IO is going to save a lot of developer time required to scale things. Using fast SSD's is probably the easiest win to speed up your database. Developer time costs a lot more than hardware and giving yourself cheap headroom up front gives you breathing room on launch.

Performance testing is also something worth doing, but the tricky part is until you roll out, it is hard to know exactly where the hotspots are going to be. In this case, new user signup would be the obvious place to test, so they probably should have tested up to the limits of their servers and tried to extrapolate an expected number of users and maybe increased that by 1.5x or something to have some leeway.

On a rollout where you don't know what you are getting into user wise, being on a cloud where you can scale out fast as demand requires is something worth doing. They could have saved a lot of bad press and headache by being on the cloud initially and migrating to less hardware after the initial peak died down.

There are a ton of little things like that you have to think about if you are dealing with massive scale. I don't know if the engineers building healthcare.gov had ever dealt with something like this before, but I'm sure they're learning these lessons now.

Re:Could Better Testing Practices Have Prevented the Healthc... by jrochkind1 on Tuesday October 08, 2013 @02:45PM (#6516986)
> if you've never had to deal with a traffic spike, you probably aren't going to build for the scale issues you will have.

One question would be whether they hired people/firms who had had to deal with that level of scale before.

I'd assume not.

But apparently they hired someone who didn't even know enough to know that they were not qualified to do it.

Re:Could Better Testing Practices Have Prevented the Healthc... by dragonwriter on Tuesday October 08, 2013 @02:50PM (#6517014)
One side effect of the complexity of government contracting processes (designed, in principle, to ensure that the government doesn't get taken advantage of by either internal or external actors) is that the people who get awarded government contracts are often the people with the most skill in negotiating the government contracting process, not the people with the most skill in the problem domain.
Re:Could Better Testing Practices Have Prevented the Healthc... by snowwrestler on Tuesday October 08, 2013 @05:05PM (#6517854)
We know who built Healthcare.gov; it was Development Seed. There were a bunch of stories about it earlier this year. Development Seed also built and runs MapBox, so they should have some idea how to run a high volume web service.
Re:Could Better Testing Practices Have Prevented the Healthc... by ericras on Thursday October 10, 2013 @11:29AM (#6528231)
And the front end built by Development Seed has had no problems. It's been the marketplace built by another contractor.
Re:Could Better Testing Practices Have Prevented the Healthc... by ErikAugust on Tuesday October 08, 2013 @12:58PM (#6516201)
"Slow DB queries are the enemy and you don't realize how bad they are until you are at scale. Sure, it only takes a few seconds on your local machine, but multiply that times thousands of concurrent users and your DB gets swamped. If you are using an ORM, it is MUCH harder to track down where in your code that 3 way join that scans every record is happening. Ideally you'd be using straight SQL and maybe use comments to tag a query. Also tools like newrelic might help if only because many databases don't offer great visibility of performance data."

1. Slow Query Log 2. Grep commands

Re:Could Better Testing Practices Have Prevented the Healthc... by camus on Tuesday October 08, 2013 @01:29PM (#6516427)
Problem is your ORM might generate 300 fast queries to get 300 records where 1 would be faster than 300. using an ORM make things tricky since you basically surrender most of Db control .

Scaling databases is hard,developpers like ORMs , but ORMs make scaling databases even harder.

Re:Could Better Testing Practices Have Prevented the Healthc... by Pxtl on Tuesday October 08, 2013 @01:57PM (#6516625)
The big thing with an ORM is that you must keep those lazy loads in mind when you work with it. Lazy loads are the big place where you get "300 queries instead of 1".

ORMs aren't bad on principal, imho... but they give you more than enough rope to hang yourself while promising a magical efficient user-friendly rope.

Re:Could Better Testing Practices Have Prevented the Healthc... by jrochkind1 on Tuesday October 08, 2013 @04:40PM (#6517695)
I wonder why most ORM's don't let you turn off lazy loads -- "only fetch when I ask for a fetch, if I try to access something without having asked you to fetch it first -- raise an exception!" Just as an option.

This would not be a particularly difficult feature for an ORM to implement.

Re:Could Better Testing Practices Have Prevented the Healthc... by Pxtl on Tuesday October 08, 2013 @07:35PM (#6518571)
Microsoft's LInQ2Entities offers this. In 3.5 it was on by default, but EF3.5 was the buggiest mess ever. They rushed it out when they realized that LInQ2SQL was unmaintainable, and then made an even worse mess.

It offered a manual "Load" operation that manually initiated the lazy load for a given FK relation, providing a worst-of-both-worlds option.

The thing I could never figure out in LinQ was how to make an explicit load of related objects fetch more aggressively. Like, say I have object Foo. I find out I need Foo->Bar, but realistically I'm also going to need Bar->Baz and Bar->Quux to make Bar useful. I could make a stand-alone query to pull down Bar, Bar->Baz and Bar->Quux in a single query, but then Bar wouldn't be loaded attached to Foo, and if I attached it to Foo it would confuse the object context. I could use the Foo->Bar to load Bar, and then Bar->Baz and Bar->Quux to load Baz and Quux... but that would be too many hits to the DB. I could never figure out how to put those things together, and fetching Foo->Bar->[Baz|Quux] proactively every time I wanted a Foo wasn't preformant.

Of course, this was back when I was obsessed with normalization and was using UUIDs for all my primary keys, so I might be missing the point.

Re:Could Better Testing Practices Have Prevented the Healthc... by jrochkind1 on Thursday October 10, 2013 @10:18AM (#6527826)
Intersting.

But to be picky, let's note that "manually initiated the lazy load" -- there's nothing "lazy" about that load, heh. The "lazy" part specifically means "on-demand, as asked for, automatically."

In ActiveRecord, you can quite easily set 'eager loading', which is setting certain loads to batch for efficiency, like you're talking about -- and this can apply to lazy loads (when you lazy load X, ALSO load y and z together, in as few SQL statements as possible), OR to manual loads (right NOW, load x, y, and z together in as few loads as possible). But AR doesn't let you turn off lazy loads, ha.

Re:Could Better Testing Practices Have Prevented the Healthc... by ErikAugust on Saturday October 12, 2013 @02:20PM (#6539677)
I see - I don't use ORMs outside of a small Django app.
Re:Could Better Testing Practices Have Prevented the Healthc... by aganders3 on Tuesday October 08, 2013 @12:57PM (#6516196)
One thing they could have done that would have saved A TON of load is not require users to sign up before giving them a list of available plans. ... My guess is millions of people just wanted to check prices and eliminating the database load for those users would have probably kept things running fast and smooth.

That's all I wanted to do (I have employer-based coverage for now). I turned back once I realized you needed an account to see this, but assumed it was due to some policy issue or agreement with the insurers.

Re:Could Better Testing Practices Have Prevented the Healthc... by jrochkind1 on Tuesday October 08, 2013 @02:47PM (#6516995)
I wonder if the list of available plans is customized, not everyone has access to every plan, and it can't show you the list of plans available to you until it knows who you are.
Re:Could Better Testing Practices Have Prevented the Healthc... by DennisP on Tuesday October 08, 2013 @02:46PM (#6516988)
What's surprising to me at this point is that nobody has published the actual plans, separately from this website. Best thing the government could do right now is get those plans and their rates out to news organizations, so people can see them without going to healthcare.gov, which could go back to being just the enrollment application that they seem to have intended.
Re:Could Better Testing Practices Have Prevented the Healthc... by shamshiel on Tuesday October 08, 2013 @03:18PM (#6517198)
The website includes the plans with the subsidies, which is why they require you to sign up and answer a bunch of questions. I believe there's a lot of verification of subsidy eligibility going on in the background. Just having the rates out there would make it appear that the plans are much more expensive than they will be for most people.
Re:Could Better Testing Practices Have Prevented the Healthc... by DennisP on Tuesday October 08, 2013 @04:54PM (#6517788)
Several news organizations have already made subsidy calculators. They just don't have specific plan details.