News for hackers. Stuff that matters.
Re:Every device with FB app is now DDoSing recursive DNS res... by littlecranky67 on Monday October 04, 2021 @05:15PM (#28752165)
Oh great, and at some point when key ISP DNS servers crack under the load, more and more websites will appear to "go down" from a users perspective - suddenly gmail.com and outlook.com not working. More and more people reload websites, restart devices etc. and increase the load even further. People fall back to using SMS/telephone, but since it is not used to that heavy load of 2021, soon phone calls fail. With FB, WA, Email and Phone "down", engineers can't be reached to fix this. And if they can, they fail to call the Uber to get them somewhere. And even if they could, the streets are congested with people that cannot communicate remotely so try to get somewhere to convey messages in-person.

Hope they will just go back to fix FB and this is just in my head :)

Re:Every device with FB app is now DDoSing recursive DNS res... by littlecranky67 on Monday October 04, 2021 @05:39PM (#28752399)
I wrote this as a pure joke, but now that I learned that SERVFAIL is not cached on browsers, clients, intermediate DNS servers [0] etc. I am curiously wondering what will be going on. It is not only FB apps, it is basically every website request (that uses FB JS for ads, tracking, etc.) that triggers a DNS request, which will be forwarded 1:1 from the ISP's DNS to the null-routed FB Subnet. This should put orders of magnitude more load on resolving DNS servers than usual.

[0]: https://serverfault.com/questions/479367/how-long-a-dns-time...

Re:Every device with FB app is now DDoSing recursive DNS res... by yourad_io on Monday October 04, 2021 @07:22PM (#28753656)
> but now that I learned that SERVFAIL is not cached on [...] intermediate DNS servers [0]

I thought so too, but at least some do cache according to Cloudflare's blog post:

> Consequently, 1.1.1.1, 8.8.8.8, and other major public DNS resolvers started issuing (and caching) SERVFAIL responses.

https://blog.cloudflare.com/october-2021-facebook-outage/

Re:Every device with FB app is now DDoSing recursive DNS res... by im3w1l on Monday October 04, 2021 @08:17PM (#28754061)
Facebook is so large that even caching the SERVFAIL with a TTL measured in seconds should cut down a lot of traffic.
Re:Every device with FB app is now DDoSing recursive DNS res... by daper on Tuesday October 05, 2021 @03:52AM (#28756556)
As I understand my favorite resolver - unbound - has separate "infra" cache that also caches RTT times, and "down" status of name servers:

https://nlnetlabs.nl/documentation/unbound/unbound.conf/#inf...

Re:Every device with FB app is now DDoSing recursive DNS res... by bowaggoner on Monday October 04, 2021 @05:50PM (#28752521)
I enjoyed the comment. You might like my short story along the same lines from a few years ago: https://bowaggoner.com/writeups/robust.html
Re:Every device with FB app is now DDoSing recursive DNS res... by pomian on Tuesday October 05, 2021 @09:52AM (#28759491)
Very fun little story. Thanks for writing it.
Re:Every device with FB app is now DDoSing recursive DNS res... by IgorPartola on Monday October 04, 2021 @05:35PM (#28752366)
Isn’t there an easy fix to just add a bullshit record for FB to DNS until this blows over?

I am also thinking that all the poorly coded sites that don’t work unless the Share on Facebook button loads are also going to hemorrhage money. So are all the e-commerce sites that rely on Login with FB.

I hope this results in everyone rethinking adding all that shit to their infrastructure before the next time.

Re:Every device with FB app is now DDoSing recursive DNS res... by jbotz on Monday October 04, 2021 @06:52PM (#28753334)
The problem isn't (wasn't) facebook.com's A records, it's that the authoritative nameservers for facebook.com are (were) unreachable. In theory someone could change the NS records for facebook on the .com nameservers to point somewhere else and serve up a fake facebook.com domain, but... 1) those NS records have a 6 hour TTL, so it would take a while be effective, and 2) who has the authority to do that?
Re:Every device with FB app is now DDoSing recursive DNS res... by DaiPlusPlus on Monday October 04, 2021 @07:22PM (#28753649)
> those NS records have a 6 hour TTL

Why on earth would any system need a TTL beyond 1 hour? The longer the TTL the longer your outage period will be if a cache is poisoned.

Re:Every device with FB app is now DDoSing recursive DNS res... by mcspiff on Monday October 04, 2021 @07:27PM (#28753705)
On the flip side, it’s the longer you can survive an outage by relying on enough clients having the cached records. So can go both ways a bit.
Re:Every device with FB app is now DDoSing recursive DNS res... by toast0 on Monday October 04, 2021 @10:44PM (#28754945)
.com (used for Facebook authoritative DNS) and .net (used for WhatsApp) servers serve glue records with a 2 day TTL, and domain holders don't get to choose TTL. (the root servers serve glue records for the TLDs with a 2 day TTL as well).

So, two days is the TTL you get for your NS records. Of course, it takes way more than two days for people to stop querying your old records. I gave up after 30-45 days.

Re:Every device with FB app is now DDoSing recursive DNS res... by jbotz on Monday October 04, 2021 @10:07PM (#28754764)
Heh. I remember the day when a TTL of a week was pretty standard.

Anyway, the since the NS records for a domain come from the next level up, i.e. in this case the .com nameserver which are also the root nameservers, if they had a short TTL then those root nameservers would get hit pretty hard. Also, it isn't your choice what the TTL on your NS records is, it's admins of your top-level domains.

Re:Every device with FB app is now DDoSing recursive DNS res... by throwaway09223 on Tuesday October 05, 2021 @12:30AM (#28755516)
"Why on earth would any system need a TTL beyond 1 hour?"

Load. Lower TTLs mean shorter term caches and more traffic.

This is a big deal, especially for root servers -- which is why the TTL is not customizable.

Re:Every device with FB app is now DDoSing recursive DNS res... by fomine3 on Tuesday October 05, 2021 @04:05AM (#28756633)
Shorter TTL increases poisoning risk. Anyway attacker can set arbitrary TTL if they succeed poisoning so longer TTL won't increase risk.
Re:Every device with FB app is now DDoSing recursive DNS res... by Thorrez on Monday October 04, 2021 @10:06PM (#28754763)
Could something with a normal 1 hour TTL be poisoned by a cache entry with a 30 day TTL?
Re:Every device with FB app is now DDoSing recursive DNS res... by dotancohen on Tuesday October 05, 2021 @01:57AM (#28755921)

  > The problem isn't (wasn't) facebook.com's A records, it's that the
  > authoritative nameservers for facebook.com are (were) unreachable.
The problem isn't (wasn't) DNS, it's that a large number of websites rely on a single point of failure in order to function properly.
Re:Every device with FB app is now DDoSing recursive DNS res... by junon on Monday October 04, 2021 @07:15PM (#28753597)
I would assume DNS providers were on call with FB staff to manually trigger flushes across the system to prop the records ASAP. Of course this doesn't affect client TTLs but resolution failures generally aren't cached anyway.
Re:Every device with FB app is now DDoSing recursive DNS res... by yourad_io on Monday October 04, 2021 @07:25PM (#28753673)
Speaking of flushing, today I learned that Google public DNS has an open "flush" function:

https://dns.google.com/cache

Re:Every device with FB app is now DDoSing recursive DNS res... by celsoazevedo on Monday October 04, 2021 @08:00PM (#28753946)
Re:Every device with FB app is now DDoSing recursive DNS res... by hamilyon2 on Monday October 04, 2021 @10:59PM (#28755037)
Dns providers are basically every ISP in existence, it would be impractical to communicate with by phone calls
Re:Every device with FB app is now DDoSing recursive DNS res... by noahtallen on Monday October 04, 2021 @06:28PM (#28752982)
My understanding is that the Facebook network itself was unreachable from the internet because of BGP. So even if an IP was resolved from DNS, that IP wouldn’t get routed to Facebook because it withdrew its routes from its peers where it connects to ISPs via BGP
Re:Every device with FB app is now DDoSing recursive DNS res... by AnimalMuppet on Monday October 04, 2021 @06:37PM (#28753127)
Not if the IP was 127.0.0.1...
Re:Every device with FB app is now DDoSing recursive DNS res... by messe on Monday October 04, 2021 @08:18PM (#28754066)
That's precisely what my ISP did. In fact, they went a step further and appear to have hijacked all unencrypted DNS queries. I was able to run DNS queries using 'example.com' as my DNS server and get a response.
Re:Every device with FB app is now DDoSing recursive DNS res... by asdff on Monday October 04, 2021 @05:43PM (#28752444)
If its any consolation, I find SMS and telephone to be remarkably robust in these situations. In college during football games, the campus population would swell to probably 200k people within a few square miles each with a cell phone in a pocket. 3g and LTE would be worthless. Campus area wifi would be worthless. The only thing that would work is shutting all that off your phone and resorting to SMS and calling people over EDGE, but it worked flawlessly even with all the people stressing out a handful of towers at once.
Re:Every device with FB app is now DDoSing recursive DNS res... by littlecranky67 on Monday October 04, 2021 @05:49PM (#28752514)
Football game's are planned events, and the carriers plan capacity accordingly. I remember sms+cell phone going down on several music festivals with 20.000+ participants (especially when it ended at around midnight). Only some carriers supported those "sponteneous" gatherings in the middle of nowhere with mobile cell towers that would keep connectivity going - but low-cost carriers never did.
Re:Every device with FB app is now DDoSing recursive DNS res... by robocat on Monday October 04, 2021 @09:29PM (#28754499)
During planned events, COWs are often deployed to provide extra mobile bandwidth. Sports areas will have design features to help (e.g. dedicated installed backhaul).

https://en.wikipedia.org/wiki/Mobile_cell_sites

I had a friend who worked deploying them many years ago to events for one of the cellphone providers in NZ.

Re:Every device with FB app is now DDoSing recursive DNS res... by ReactiveJelly on Monday October 04, 2021 @08:23PM (#28754101)
It would be cool to see Matrix's 100bps experiment allow full apps (with E2EE and other modern features, as opposed to SMS) to handle horrid network conditions.

https://matrix.org/blog/2019/03/12/breaking-the-100-bps-barr...

(I think this was posted already, and it's from 2019, but some people must have missed it - I only saw it this year)

Re:Every device with FB app is now DDoSing recursive DNS res... by mschuster91 on Monday October 04, 2021 @06:30PM (#28753036)
German carrier O2 was/is notorious for offering somewhat decent-ish service in urban areas under normal conditions - but major events that happen to have lots of people moving around the city like fans congregating to a soccer game with public transport, political rallies or your average drunkard festival (=Oktoberfest)? Instant collapse...
Re:Every device with FB app is now DDoSing recursive DNS res... by tarsius on Monday October 04, 2021 @06:06PM (#28752682)
For those you understand Swiss German: Mani Matter – I han es Zündhölzli azündt https://www.youtube.com/watch?v=PkGatIgXERI
Re:Every device with FB app is now DDoSing recursive DNS res... by Tainnor on Monday October 04, 2021 @06:53PM (#28753345)
I lit a match / And it made a fire. / And for my cigarette / I wanted to take the fire from the match. / But the match slipped from my hand and landed on the rug. / And it almost made a hole in the rug.

Well, you know what can happen / If you're not careful with fire. / And for the light on a cigarette / A rug feels rather too expensive. / And from the rug the fire, alas, / Might have spread to the whole house / And who knows what would have happened thereafter?

There would have been a fire in the district / And the fire fighters would have had to come. / They would have honked in the streets / And unloaded their pipes. / And they would have sprayed fire / And it would have been in vain / And the whole city would have burned with nothing to protect it.

And the people would have jumped around / Fearing for their possessions. / They would have thought somebody had started a fire. / They would have grabbed assault rifles. / Everyone would have shouted: "Whose fault is it?" / The whole country would have rioted. / And they would have shot at the ministers behind the lecterns.

The UN would have become involved / And the UN enemies as well. / To preserve peace in Switzerland / Both would have come with tanks. / It would have spread, little by little, / To Europe, to Africa. / There would have been a World War and humans would be no more.

I lit a match / And it made a fire. / And for my cigarette / I wanted to take the fire from the match. / But the match slipped from my hand and landed on the rug. / Thankfully, I picked it back up.

Re:Every device with FB app is now DDoSing recursive DNS res... by BiteCode_dev on Tuesday October 05, 2021 @05:48AM (#28757211)
It's already the case. It's been some times since some websites randomly are hard to reach for me. Changing DNS helped, but the best move has been a VPN to a far away country.