News for hackers. Stuff that matters.
Re:Issues with upstream DNS provider by rajamaka on Tuesday August 23, 2022 @07:31PM (#32572997)
Production completely inaccessible, on the plus side I might finally now get backing to migrate away from the Heroku cesspit.
Re:Issues with upstream DNS provider by bobsingor on Tuesday August 23, 2022 @07:49PM (#32573181)
Yes, i'm also done with Heroku. What alternative do you recommend?
Re:Issues with upstream DNS provider by benschwarz on Tuesday August 23, 2022 @09:56PM (#32574140)
My company moved a 600GB postgres db to crunchydata (https://www.crunchydata.com), the process was super smooth. In fact, it went so well that I agreed to a quick case study: https://www.crunchydata.com/case-studies/calibre.

We're currently using Heroku for app hosting, while evaluating fly.io, render.com and railway.app. All three have had exceptional reviews from other customers and differ slightly on their service offerings and setups. All seem like viable alternatives so far!

Re:Issues with upstream DNS provider by rsavage on Wednesday August 24, 2022 @07:31AM (#32577931)
I’d be really interested to know how your research goes. We are running a large production scale app on heroku. And that we have no DevOps or Infra staff is awesome. But heroku has been a huge let down lately.

Most of the reviews we have seen of the competitors are all hobby level. And last time we check some of this competitors we found their security posture was not the level we would require.

So we had to simple rule them out and either stay with Heroku or move to a big 3.

If you’re keen to share - let me know and I’ll send you my details.

Re:Issues with upstream DNS provider by craigkerstiens on Wednesday August 24, 2022 @10:08AM (#32579654)
Craig here from Crunchy Data. I think you're speaking to the app side of things on hobby level. On the database side of things our security posture for Crunchy Bridge I'd say is stronger than the Heroku one. By default we isolate all databases in a VPC, everything is purely single tenant where as Heroku Postgres at least when I was there had multiple forms of multi-tenancy which when doing multi-tenancy in Postgres can have risks[1]-this applies even to the major 3 cloud providers. Our team is essentially the original Heroku Postgres team so we've built with security but also user experience for Postgres in mind since day one.

Now I assume you were speaking to the 3 mentioned, render, railway, fly in terms of hobby level. All three are fairly young relative to Heroku's age, but Fly did recently get their SOC2 and the team really took it to heart and invested in it so I'd put some stock in that. I can't speak definitively to the others, but do know all three can be solid for production apps. If you've got HIPAA or other specific requirements I'd encourage a conversation with them.

[1] https://www.wiz.io/blog/the-cloud-has-an-isolation-problem-p...

Re:Issues with upstream DNS provider by rsavage on Friday August 26, 2022 @08:55AM (#32606812)
Thanks Craig. I took at look at Crunchydata - however best I can tell unless we are in Enterprise Tier Heroku (or maybe not even then) we have to connect to Crunchydata via internet (with IP whitelist?) rather than through VPC peering or similar. Which is a limitation of Heroku rather than you. I assume with something like fly it could be done via VPC peering?

I just read fly had SOC2 type I recently. But I mean this hosting infra containing all our data and our customers data. People providing infra really need to take security extremely seriously and prove it.

Awesome what they are doing - just don’t feel like they are ready for primetime busines. We are a small startup (5k monthly on Heroku) but there is just no reasonable way we can tell our enterprise customers security teams are hosted on these guys and can vouch and vet their security.

Once fly has type II - we’ll take another look.

Re:Issues with upstream DNS provider by bennylope on Wednesday August 24, 2022 @04:55PM (#32585620)
This is the same path we're on. Migrated to Crunchy a month ago or so to remove the major migration risk and are using Render to host an auxiliary service while our core application remains on Heroku. Haven't yet done any non-toy deployments on Fly.io or Railway but I very much like Render's Blueprints and environment groups.
Re:Issues with upstream DNS provider by alexthehurst on Wednesday August 24, 2022 @12:35AM (#32575039)
For what it's worth: Crunchydata appears to have been using Heroku DNS in some way. We have a Heroku app with Crunchy databases, and our Crunchy dbs became inaccessible shortly before the app did.
Re:Issues with upstream DNS provider by craigkerstiens on Wednesday August 24, 2022 @10:08AM (#32579661)
Craig here from Crunchy, we do run some small pieces on Heroku, but during the DNS outage saw no interruption of databases and they all seemed to be available and up. Please do feel free to drop us a note and we'll happily investigate if something did occur there.
Re:Issues with upstream DNS provider by caffeinated_me on Tuesday August 23, 2022 @08:22PM (#32573446)
We did some trials and ended up on Crunchy Data for the Postgres part of the equation. Wish we had done so earlier, logical replication was a big win.

Sadly, we're still using Heroku DNS, but this should accelerate finding an alternative.

Re:Issues with upstream DNS provider by tortilla on Tuesday August 23, 2022 @07:50PM (#32573184)
I'm looking at fly.io and render.com.
Re:Issues with upstream DNS provider by fourstar on Tuesday August 23, 2022 @08:06PM (#32573321)
I moved from Heroku to Render. Can't recommend them enough. Smaller operation, of course so keep that in mind if you need support, but for a hobbyist like myself, I'm running 2 postgres dbs, 1 redis instance, and 4 web services (2 websites).
Re:Issues with upstream DNS provider by anurag on Tuesday August 23, 2022 @10:13PM (#32574226)
(Render founder) thanks for the rec and feel free to get in touch with me anytime (email in profile). For those who need them, we offer enterprise support plans with response and uptime SLAs.
Re:Issues with upstream DNS provider by dogweather on Tuesday August 23, 2022 @09:48PM (#32574083)
I just completed the move of my web app to render, and for now connecting back to Heroku for the postgres instance. I'll move that over eventually.
Re:Issues with upstream DNS provider by camjohnson26 on Tuesday August 23, 2022 @07:56PM (#32573235)
We’re using Porter to simplify the migration to AWS
Re:Issues with upstream DNS provider by mtlynch on Tuesday August 23, 2022 @07:58PM (#32573261)
Left Heroku for Fly.io a year or so ago and have been very happy.
Re:Issues with upstream DNS provider by nicoburns on Tuesday August 23, 2022 @08:51PM (#32573658)
How have you found uptime on fly.io (and are you using multiple instances for a high-availability setup?). We've been running on Heroku without HA, and it's been good enough, and wondering if we might expect to be able to get away with that on fly.io too...
Re:Issues with upstream DNS provider by mtlynch on Tuesday August 23, 2022 @10:16PM (#32574237)
Uptime has mostly been good, although I didn't set up alerting for any of my Fly projects until recently. All my apps are running on a single node per app, so I haven't tried

The one notable outage was this bug where Fly was evicting smaller instances at overloaded DCs rather than preventing new apps from acquiring resources:

https://community.fly.io/t/app-stuck-in-pending-state-after-...

The workaround was to upgrade to a larger instance, but I probably evicted someone else.

Re:Issues with upstream DNS provider by ab-dm on Tuesday August 23, 2022 @08:30PM (#32573512)
Did you use the https://fly.io/launch/heroku migration page? It almost looks too good to be true.
Re:Issues with upstream DNS provider by mtlynch on Tuesday August 23, 2022 @10:10PM (#32574216)
No, I wasn't using much Heroku-specific. All of my apps run in Docker containers, so it's mostly just migrating my CI to deploy to Fly instead of Heroku.

I did get a lot of perf increases moving from GCP to Fly, but I think I have to credit most of the improvements to SQLite being faster than Firestore:

https://mtlynch.io/retrospectives/2021/12/#migrating-my-side...

Re:Issues with upstream DNS provider by Exuma on Tuesday August 23, 2022 @08:58PM (#32573730)
wow it def does lol
Re:Issues with upstream DNS provider by geoffharcourt on Wednesday August 24, 2022 @05:51AM (#32577147)
We moved our app to Porter after the credentials incident this spring and have been very pleased.
Re:Issues with upstream DNS provider by rajamaka on Tuesday August 23, 2022 @07:54PM (#32573218)
Moving to ECS on AWS, only reason is that I'm a DevOps engineer with familiarity in that space. Honestly wouldn't know where to send developers without the time to figure the nuts and bolts of convenience that Heroku provides.
Re:Issues with upstream DNS provider by zoomzoom on Tuesday August 23, 2022 @08:09PM (#32573350)
Please check out withcoherence.com (I'm a cofounder). We manage the nuts and bolts of apps using ECS/fargate on AWS and Cloud Run/GKE on GCP. We also integrate managed cloud IDEs and managed CI/CD pipelines using the same simple configuration, and put a dashboard on top of it!
Re:Issues with upstream DNS provider by dogweather on Tuesday August 23, 2022 @08:55PM (#32573698)
Render.com has been great. Our test site is up, connecting back to Heroku for postgres. ( https://www.public.pink )

I just happened to try out Render last week - perfect timing.

Re:Issues with upstream DNS provider by tortilla on Tuesday August 23, 2022 @07:32PM (#32573012)
Yep...it's time (me too).