News for hackers. Stuff that matters.
Re:Rearchitecting GitHub Pages by charliesome on Wednesday May 27, 2015 @11:37AM (#9612106)
We do have Fastly in front of *.github.io, but there's still a significant amount of traffic (on the order of thousands of requests per second throughout the day) that make it through to our own infrastructure.

We don't do any other caching on our own, although the other replies are correct in that the Linux kernel has its own filesystem cache which means not all requests end up hitting the SSD.

Re:Rearchitecting GitHub Pages by simonw on Wednesday May 27, 2015 @02:34PM (#9613417)
Do you explicitly invalidate Fastly, or do you use a timeout? If so, how long is the timeout?
Re:Rearchitecting GitHub Pages by kyledrake on Wednesday May 27, 2015 @04:02PM (#9613935)
Invalidating quickly is the main problem with caching on a CDN. We use an async worker via Redis pubsub to call for expiration on individual files locally on proxy servers we run. I'm looking at using NSQ for this in the future. One interesting solution is to just use an HTTP hit to expire a cache, which you can see a flavor of in our nginx config file. Nginx in effect becomes it's own cache SoA. We needed a special nginx module to make that work, default nginx only lets you expire the entire cache.

Fastly probably has something similar. You've really got to do this within 5 seconds or your user is going to get pissed off waiting for it every time they save/reload the page.

The big problem with passing caching to third party CDNs is that they need to be able to handle your SSL certs inline to request static files you don't need to change the URL of (because then you would be changing the published content which you don't control), and supports wildcards.

In effect you're doing what Cloudflare does. I'd use Cloudflare in a minute for this, but wildcards require their commercial plan and we can't afford it ($6k/mo). Also we would need fine-grained cache expire.

If Fastly does this and is priced in our range, maybe we should talk to them. :)

Re:Rearchitecting GitHub Pages by aspir on Wednesday May 27, 2015 @04:22PM (#9614061)
FWIW Fastly invalidates objects, services, and batches of objects globally at ~150ms via an API. They have a different approach to wildcard/batch invalidation: it groups based on a prior-set header that can be programmatically based off of URL paths (or a bunch of other things).

Disclaimer: I'm employed by Fastly :)

Re:Rearchitecting GitHub Pages by kyledrake on Wednesday May 27, 2015 @04:29PM (#9614105)
That would work for Neocities. Wildcard pricing is unfortunately a little out of our budget, but if you were interested in coming to an arrangement with us (we could say powered by Fastly on the site, for example) in exchange for a reduction on that, let me know. :) [email protected]
Re:Rearchitecting GitHub Pages by icefox on Wednesday May 27, 2015 @01:30PM (#9612938)
Are the assets extracted from git and dumped on the filesystem or are they fetched from the Git objects? aka If I generate a git tree object with millions of files, but only pointing to a handful of blobs (think a maze) would this exploded on the filesystem?
Re:Rearchitecting GitHub Pages by fugyk on Wednesday May 27, 2015 @12:26PM (#9612461)
What percentage of the requests is handled by CDN?
Re:Rearchitecting GitHub Pages by jsingleton on Wednesday May 27, 2015 @12:50PM (#9612651)
I believe that only CNAMEs use the CDN for custom domains (naked domains do not) [0]. So that rules out a chunk but there will be lots of cache misses too. Maybe someone from GH can confirm.

[0] https://help.github.com/articles/about-custom-domains-for-gi...

Re:Rearchitecting GitHub Pages by michaelmior on Wednesday May 27, 2015 @03:38PM (#9613788)
github.io will still also hit the CDN. It's just custom naked domains that will miss.
Re:Rearchitecting GitHub Pages by toomuchtodo on Wednesday May 27, 2015 @03:23PM (#9613692)
Their docs mention ever 200 response code request is cached, so I'd say a majority of the requests?