News for hackers. Stuff that matters.
Re:Netlify CDN Outage by epoch_100 on Monday November 04, 2019 @01:23PM (#21444631)
Netlify is describing the outage as being the result of some kind of attack:

"We’re currently suffering an attack that’s affecting our Free and Self-Serve CDN. Enterprise is currently unaffected. We are working on mitigating the effects and will update with more information as it becomes available."

Re:Netlify CDN Outage by chrisblackwell on Monday November 04, 2019 @12:56PM (#21444296)
I understand the growing pains Netlify must be going through. They have created a great product, and are becoming very popular. I'd love to see the Enterprise version (or any paid version) segregated out from the free sites so we can get more reliable uptime.

The build pipeline was down for several hours a few days ago, now a CDN outage makes things much worse, as it's taken down all our sites.

I'm sure they will resolve things quickly, in the meantime our clients will start asking about other failover options.

Re:Netlify CDN Outage by drewda on Monday November 04, 2019 @01:14PM (#21444529)
According to their overall status page, they already do run two separate CDNs for normal and enterprise customers: https://www.netlifystatus.com/
Re:Netlify CDN Outage by luhn on Monday November 04, 2019 @01:15PM (#21444533)
Ostensibly they do seem to have that.

> Our non-enterprise CDN is failing a small number of page loads, usually with SSL errors; Enterprise CDN is not affected.

But looks like there's room for improvement, because the enterprise CDN ended up failing as well.

> We are still working to mitigate the effects of these problems and do see some page load failures on the enterprise CDN as well.

Re:Netlify CDN Outage by gravypod on Monday November 04, 2019 @01:00PM (#21444334)
It'd be really cool to see a writeup of what is going on at Netlify.
Re:Netlify CDN Outage by nwsm on Monday November 04, 2019 @01:20PM (#21444595)
They are tweeting about it, and I'd expect a post mortem as they have been transparent in the past.
Re:Netlify CDN Outage by sergiotapia on Monday November 04, 2019 @02:42PM (#21445584)
So I wasn't going crazy haha - just today we migrated our site to Netlify from Heroku and the second I wanted to demo it my boss the COO, it was failing and not loading.

It wasn't just netlify though. I also was having trouble loading random websites like aptible, youtube, imgur.

Re:Netlify CDN Outage by swaraj on Monday November 04, 2019 @02:15PM (#21445280)
This affects us too. We were able to migrate to zeit now in ~30 mins.
Re:Netlify CDN Outage by ngrilly on Monday November 04, 2019 @02:33PM (#21445479)
Will you migrate back to Netlify when Zeit will suffer a similar downtime?
Re:Netlify CDN Outage by swaraj on Monday November 04, 2019 @02:56PM (#21445750)
We're definitely staying on Netlify after this outage is done. We pay for some of their enterprise features, e.g. audit logs. We just need our sites up right now and zeit was a fast, temporary alternative
Re:Netlify CDN Outage by ngrilly on Monday November 04, 2019 @03:03PM (#21445847)
Do your DNS records have a short TTL to enable such a quick switch from one provider to another?
Re:Netlify CDN Outage by swaraj on Monday November 04, 2019 @08:33PM (#21448769)
Fortunately (or non-optimally), our DNS TTLs were 300s (Route 53 default) so our sites were not fully accessible off Zeit for ~5 mins. This wasn't too bad given that our sites were completely inaccessible from Netlify.

One day when we're bigger, we'll extend those TTLs

Re:Netlify CDN Outage by ngrilly on Tuesday November 05, 2019 @03:05AM (#21450526)
Short TTLs are bad, but I confess using them in some small scale projects in anticipation of this kind of worst case scenario :)
Re:Netlify CDN Outage by nwsm on Monday November 04, 2019 @01:13PM (#21444516)
My site is back up. (1:13 EST)

EDIT: intermittent issues now. Sometimes nothing loads, sometimes everything, sometimes SSL issue, sometimes HTML but no CSS or JS load. (1:22 EST)

Re:Netlify CDN Outage by alfredxing on Monday November 04, 2019 @02:49PM (#21445678)
Their CDN hostname, `cdn.netlify.com`, resolves to a DigitalOcean node, so it seems like they are rolling their own instead of delegating to a third party.
Re:Netlify CDN Outage by tschellenbach on Monday November 04, 2019 @01:00PM (#21444345)
Yup, Stream's blog is down as well. Thought it was WPEngine at first, but turns out it's netlify.
Re:Netlify CDN Outage by losvedir on Monday November 04, 2019 @01:21PM (#21444602)
Huh, loaded HN because it seems YouTube videos are down for me. I wonder if it's related.
Re:Netlify CDN Outage by nwsm on Monday November 04, 2019 @01:23PM (#21444633)
Another comment claimed that Netlify rents servers from other providers, so this could be the case.

FWIW Youtube is fine for me right now.

Re:Netlify CDN Outage by volkk on Monday November 04, 2019 @01:57PM (#21445048)
insane coincidence but literally the minute i started my first project ever on netlify just to play with it, i deployed it and nothing was loading; i kept googling and thinking it was my fault and then it clicked to check their status page and alas.
Re:Netlify CDN Outage by jagannathtech on Monday November 04, 2019 @09:54PM (#21449176)
So it was YOU who took down the network then
Re:Netlify CDN Outage by ajsharp on Monday November 04, 2019 @01:07PM (#21444434)
netlify.com appears to be affected as well.

Curious if this is a software issue or a host issue. I believe netlify rents a lot of its infrastructure from other providers, such as Digital Ocean.

Re:Netlify CDN Outage by cfors on Monday November 04, 2019 @01:14PM (#21444525)
At any sort of scale (which I have to imagine netlify is running at), this will be a software issue since hosts are designed to fail.

Serving static content is crazy easy for computers, my guess is a bad configuration got pushed out.

Re:Netlify CDN Outage by altmind on Monday November 04, 2019 @01:37PM (#21444815)
Here I saw some mentions that it may be related to recent google cloud problems https://news.ycombinator.com/item?id=21418979
Re:Netlify CDN Outage by twobat on Monday November 04, 2019 @01:59PM (#21445067)
Bad publicity is still publicity, right?