Heroku - if you're listening. Please do NOT migrate IPs / load balancers here for a little while. You might also do some work to detect those of us who have bypassed your DNS, and give us some nudges individually to move back to your DNS when the time is right.

You just caused a very painful outage for your entire customer base, and a number of us used this hack to get back online. If you don't recognize this, and cause ANOTHER outage (the third in the past few weeks...) we're going to flee even faster to more stable platforms.

Those IPs tend to stay pretty stable and are shared across the fleet. While you should change them back once things are stable, it's reasonably likely these won't change out from under you in a few hrs.

Hi all, deleting a root CNAME record and adding a new A record on Cloudflare was too risky for us, with its potential to impact external DNS responses, so we took the following steps instead:

1. Create a new temporary subdomain in the Cloudflare DNS panel, e.g. "temp-heroku-resolve.your-domain.com", with IPs from the responses you get by querying Heroku's upstream provider directly (`dig @1.1.1.1 +trace your-app.herokuapp.com`). Make them unproxied (grey cloud) A records with a 30s TTL.

2. Change your root CNAME to point at "temp-heroku-resolve.your-domain.com"

To undo, just reverse these steps and point your root CNAME back at Heroku.

This is essentially doing the same thing right? What is the advantage of this method? Less DNS changes?

(not trying to be snarky, trying to understand)

The advantage is that you can do the CNAME swap atomically in the Cloudflare interface, instead of deleting a CNAME record and adding a new A record, which would lead to a few seconds when your DNS is entirely removed. Those few seconds may not sound like a lot, but they could be cached by Cloudflare or other external DNS resolvers, trigger "clean-up" functions within Cloudflare to remove resources associated with those domains, etc.

In our case, we have substantial logic running on the edge in Cloudflare Workers, as well as lots of other Cloudflare configuration options, and I don't even want to know what it could potentially do to our zone to remove the root DNS record that everything's tied to.

Sounds like cloudflare could add an atomic cname -> a record feature.

Cloudflare should support RFC 2136 which includes atomic updates and is the standard for DNS updates.

Honestly, I think this solution worked better than any dedicated feature could have, since it used the exact same existing codepaths that Cloudflare already had for CNAME flattening. Using CNAME records for self-documentation seems like a better practice anyway. Spending the time building a complicated new feature for such a rare case seems like a bad idea.

Good points, thanks!

Deleting a record causes requests for that record to be negatively cached. The negative cache TTL[1] is often set to 1 day. So, deleting and immediately recreating a record can take your site offline for days. (Source: I made this mistake when changing a Route53 record for a subdomain that was getting about 70Krps. Luckily it was our source code making the requests, so we could change it, but that took an hours or so to roll out.)

[1]: https://www.rfc-editor.org/rfc/rfc2308

Nah bc you're not swapping from CNAME > A records, which could have other random networking related impacts, especially if you're running a public API.

Nice. We got our test site online by migrating just the dyno to render.com. (And connecting back to Heroku for the database.) We might just switch our DNS there and call it a permanent move.

probably worth staying at render considering how many issues heroku's had recently

Anybody know what the risks are for doing this? We have not one but 13 heroku domains running where we might do this.

What are the chances that their current work on the DNS issues might replace the underlying IP-addresses and hence we are making it harder on ourselves?

I thought about this, and it can't really get any worse than 'total outage', so I thought it was worth taking a shot. It can't go any more down.

It worked for me, so once Heroku has sorted their s*t out and their DNS system is back up it's just a matter of deleting the A records and replacing it with the CName.

Worst case scenario is the underlying Amazon IP Addresses change and it goes back down.

Wouldn't the worst case be the underlying IP address changes, and your site now points to some other service? Since Heroku doesn't offer static IPs, I don't think.

Heroku seems to use the Host header on the HTTP request to determine what customer the request is sent to, so I don't think it can end up somewhere it shouldn't be unless the IP leaves control of Heroku entirely and is picked up by another AWS customer.

Heroku uses the host name to route requests. So you wouldn't serve another site, it would just give you an error that your site wasn't found or something.

There is one more problem I can think of (but yes you're right, it can't get worse than 'total outage'): Heroku might detect that the CNAME is missing and will deactivate some stuff from their automatic domain management / TLS certificate issuing.

Oh my god, thank you so much. You're a lifesaver.

For those on Cloudflare, this will still work. I just deleted the existing CNAME, added the 4 IP addresses as seperate A records and it came back up instantly.

No historical DNS records on that site for us unfortunately, any other possible sites to check?

Try this command: nslookup DOMAIN 212.230.255.1

That is a random public DNS server I found on a DNS checker site that works (as it seems to be down from all the common ones like 8.8.8.8, 1.1.1.1, etc). DOMAIN can be your actual domain or the long heroku alias.

Thanks, for some reason this worked!

You bloody legend, thanks!

you saved my users! thanks so much

This one worked for me: https://dnshistory.org/

Used this site https://www.whatsmydns.net, and it worked.

Unfortunately the Heroku issue lost us a few thousand $ alongside a few customers.

This worked for us, thank you.

And for those using Cloudflare, this method works.

This got us back online. Thank you so much.

Can someone clarify what to do if your app doesn't use a custom domain, and running `heroku domains -a` just returns `my_app.herokuapp.com`, and then entering that into https://securitytrails.com/dns-trails returns no results?

Do you use Cloudflare?

No we just run our API on heroku and use no custom domain names. For some of us requests are working as normal, but for others every request is failing.

What error are you getting? Because for us it seemed that the Cloudflare <-> HerokuDNS link was down. But even then I was able to access my app from the dashboard (like https://my-app.herokuapp.com/)

Presumably if your site is behind Cloudflare then this strategy won't work, right? Since the IP addresses that Security Trails sees are just of Cloudflare rather than your actual Heroku IPs...?

Or is it possible in the Cloudflare dashboard there is somewhere to see your Heroku server's IP address?

You need to put in the herokudns.com address that the CNAME is pointing at – e.g. stark-wisteria-rnbgkawldfk6gq7m8308ytts.herokudns.com in our case.

For me the securitytrails.com website just crashes. I put my DNS target: "stark-horse-mrp4jeowu9yvwpnnma32x6hd.herokudns.com", clicked "Run Check" and it seems to redirect to a failed (no CSS) webpage. Anyone else experiencing this?

EDIT: I MANAGED to make it work with this:

https://www.nslookup.io/

Make sure to go through all the tabs at the top (Cloudflare DNS, Google DNS) - for me they were all "no A records found". Only "Authoritative" gave me 3 A records which I successfully managed to use.

That seems to happen when you put in a DNS target that doesn't have any records in SecurityTrails. In that case, it is best to use nslookup

Yeah securitytrails.com was working for me, but went down about 10 mins ago.

Thanks so much, it worked for us.

We use cloudflare. If you look up the IP address for your public domain name, you will get the cloudflare IP, yes. If you lookup the IP address of the CNAME target, you will get the heroku IP.

on Cloudflare do you create A records (with found IPs) with name "www" or "mydomain.com"? also do you make that A record proxied or no? Thank you!

For my domain (https://www.poof.io), it's www.

If you just use something like https://poof.io, then it would be @. Depends on your site.

There should be a few historical IP addresses, but you would create an A record for each of them.

Use an A record and type in @. Proxied is fine.

Worked for us on Cloudflare

We use Cloudflare + Heroku and it worked for us.

It worked for us too. Thanks.

Note: If you used to have a CNAME record for yourdomain.com to www.yourdomain.com, then you have to add two A records per each IP (one whose name is yourdomain.com pointing to the IP, and another whose name is www pointing to the same IP)

Bless you! Thank you for taking the time out of your day to help us internet people :)

Heroku reports the outage is over - but I wasn't able to resolve our Heroku-based domain name until recently today.

Should be able to go back to CNAME now. Regardless, recheck yours

Worked... thanks a bunch. Does anyone know if it's safe to go back to cname now or should I leave till the A record till the morning? Would prefer to get some sleep.

Worked for us on Cloudflare using the .dns Heroku target!

Thanks so much, worked for us as well. I created A records for all three of the listed IP addresses but it started working for the first one already.

Yes, this worked for me...I was trying to do this right away, but I was missing a way to find historical DNS, thank you!!

Woot! Thank you - that seems to work, just used the first IP that came up on ST (there were 4 listed).

This saved our bacon and got me a pat on the back from our VP. Thanks!

You and your team are brilliant. Thank you for this post! Much love.

This worked for us. We're using Heroku + Cloudflare.

Anyone know how to make this work for a subdomain?

Do you have a CNAME record pointing to a herokudns.com domain? Replace it with an A record that points to the IP instead. You can find the IP by going to https://securitytrails.com/domain/<something-1234.herokudns....

Wow this worked! Your amazing! Thank you so much

I think I love you. We're back online.

Worked for me as well! Thank you!

Thank you! You saved our ass.

Thank you a lot!!

This worked for us too - thank you!

Edit: It also works for subdomains.

Working well !