News for hackers. Stuff that matters.
Re:Rearchitecting GitHub Pages by jrochkind1 on Wednesday May 27, 2015 @11:28AM (#9612042)
All of github pages was run off of _one_ server? (with one failover standby).

That's pretty amazing. If all you're serving is static assets, apparently you have to grow to pretty huge scale before one server will not be sufficient.

I'm curious if there was at least a caching layer, so every request didn't hit the SSD. They didn't mention it.

Re:Rearchitecting GitHub Pages by charliesome on Wednesday May 27, 2015 @11:37AM (#9612106)
We do have Fastly in front of *.github.io, but there's still a significant amount of traffic (on the order of thousands of requests per second throughout the day) that make it through to our own infrastructure.

We don't do any other caching on our own, although the other replies are correct in that the Linux kernel has its own filesystem cache which means not all requests end up hitting the SSD.

Re:Rearchitecting GitHub Pages by simonw on Wednesday May 27, 2015 @02:34PM (#9613417)
Do you explicitly invalidate Fastly, or do you use a timeout? If so, how long is the timeout?
Re:Rearchitecting GitHub Pages by kyledrake on Wednesday May 27, 2015 @04:02PM (#9613935)
Invalidating quickly is the main problem with caching on a CDN. We use an async worker via Redis pubsub to call for expiration on individual files locally on proxy servers we run. I'm looking at using NSQ for this in the future. One interesting solution is to just use an HTTP hit to expire a cache, which you can see a flavor of in our nginx config file. Nginx in effect becomes it's own cache SoA. We needed a special nginx module to make that work, default nginx only lets you expire the entire cache.

Fastly probably has something similar. You've really got to do this within 5 seconds or your user is going to get pissed off waiting for it every time they save/reload the page.

The big problem with passing caching to third party CDNs is that they need to be able to handle your SSL certs inline to request static files you don't need to change the URL of (because then you would be changing the published content which you don't control), and supports wildcards.

In effect you're doing what Cloudflare does. I'd use Cloudflare in a minute for this, but wildcards require their commercial plan and we can't afford it ($6k/mo). Also we would need fine-grained cache expire.

If Fastly does this and is priced in our range, maybe we should talk to them. :)

Re:Rearchitecting GitHub Pages by aspir on Wednesday May 27, 2015 @04:22PM (#9614061)
FWIW Fastly invalidates objects, services, and batches of objects globally at ~150ms via an API. They have a different approach to wildcard/batch invalidation: it groups based on a prior-set header that can be programmatically based off of URL paths (or a bunch of other things).

Disclaimer: I'm employed by Fastly :)

Re:Rearchitecting GitHub Pages by kyledrake on Wednesday May 27, 2015 @04:29PM (#9614105)
That would work for Neocities. Wildcard pricing is unfortunately a little out of our budget, but if you were interested in coming to an arrangement with us (we could say powered by Fastly on the site, for example) in exchange for a reduction on that, let me know. :) [email protected]
Re:Rearchitecting GitHub Pages by icefox on Wednesday May 27, 2015 @01:30PM (#9612938)
Are the assets extracted from git and dumped on the filesystem or are they fetched from the Git objects? aka If I generate a git tree object with millions of files, but only pointing to a handful of blobs (think a maze) would this exploded on the filesystem?
Re:Rearchitecting GitHub Pages by fugyk on Wednesday May 27, 2015 @12:26PM (#9612461)
What percentage of the requests is handled by CDN?
Re:Rearchitecting GitHub Pages by jsingleton on Wednesday May 27, 2015 @12:50PM (#9612651)
I believe that only CNAMEs use the CDN for custom domains (naked domains do not) [0]. So that rules out a chunk but there will be lots of cache misses too. Maybe someone from GH can confirm.

[0] https://help.github.com/articles/about-custom-domains-for-gi...

Re:Rearchitecting GitHub Pages by michaelmior on Wednesday May 27, 2015 @03:38PM (#9613788)
github.io will still also hit the CDN. It's just custom naked domains that will miss.
Re:Rearchitecting GitHub Pages by toomuchtodo on Wednesday May 27, 2015 @03:23PM (#9613692)
Their docs mention ever 200 response code request is cached, so I'd say a majority of the requests?
Re:Rearchitecting GitHub Pages by vog on Wednesday May 27, 2015 @11:31AM (#9612068)
I believe this is automatically done by the operating system. Nginx usually performs some kind of mmap of the requested file, and the virtual memory management will automatically take care of caching.

BTW, I'm often surprised how people are afraid from opening files from their scripts, as if they think this will always lead to disk access. Then, they start implementing a hand-written caching layer on top of that, which usually performs worse than what the OS already offers.

Re:Rearchitecting GitHub Pages by falcolas on Wednesday May 27, 2015 @12:49PM (#9612645)
If it's a static file with no dynamic compression (nginx allows the use of pre-compressed assets) or SSL, nginx doesn't even have to open the file, it just uses the sendfile system call.
Re:Rearchitecting GitHub Pages by shanemhansen on Wednesday May 27, 2015 @02:35PM (#9613420)
Minor nit. The sendfile system call requires an open file. nginx can just skip reading the file contents into userspace.

Also, BSD has some recent extensions to make sendfile work with ssl [https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf] (pdf)

Re:Rearchitecting GitHub Pages by kyledrake on Wednesday May 27, 2015 @03:54PM (#9613886)
Correct. Static file serving is amazingly efficient. Even the filesystem caches for you automagically with whatever RAM you're not using. There's even a kernel function for speeding up file transfers, that's how optimized this stuff is: http://man7.org/linux/man-pages/man2/sendfile.2.html

My rough estimates are that Neocities can handle 20-50 million sites with only two fileservers and a sharding strategy. So double it for each shard, and you've got a solution that actually scales pretty well.

Re:Rearchitecting GitHub Pages by bkeroack on Wednesday May 27, 2015 @04:36PM (#9614147)
It's not that surprising. Serving static files is pretty easy and lightweight (part of why I like the movement towards SPAs), and there's been 20+ years of work done to make it fast. We've been conditioned for years by dynamic application servers that perform multiple DB queries, external service calls, etc per request. It doesn't have to be slow or inefficient.
Re:Rearchitecting GitHub Pages by nomel on Wednesday May 27, 2015 @11:32AM (#9612074)
SSD cache and os file system cache is always there. I'm sure they were filled with ram, and that's all it would be used for.