Heroku Shield is a service added on top of Heroku Private spaces.
You usually don’t need Shield unless you want to be compliant with things like HIPAA, etc
Which of course could be your case here.
I've always been curious. What do you need to do to be HIPAA compliant, from a technology standpoint? I figured it's similar to PCI compliance, but I'm not sure.
From what I've heard, though, the cost isn't quite zero, it's just that you have to own & implement all the work to be HIPAA compliant. But perhaps it's not that bad?
I know that for a customer I spoke to, keystroke logging on running dynos was something they were really interested in, from a compliance point of view.
I think being able to spin up Postgres DBs with rollbacks, fork and follow, HA etc etc (don’t want to sound like a sales rep) in this highly compliant environment also involves some serious infra wrangling.
We were about to use Heroku Shield at a previous gig. It's definitely expensive, but at our requirements it was still less than an engineer. I wouldn't run a ton of "big data" processing on Heroku nodes though. I'm sure/hope it exists, but I haven't seen a Heroku-ized version of data processing.
Author here; I don’t want to go into too much detail, but we tried Shield early on and had a negative experience that made us wary about using the platform (it seems to use a different tech stack under the hood from “normal” Heroku and lacks a lot of the things that make Heroku great). Also it’s very expensive compared to VPC-based solutions on AWS and GCP.
W.R.T. the batch jobs, I think I didn’t explain super well—we are using a different language and runtime from our “normal” background processing jobs (which use worker queues in Rails), it’s just that Heroku isn’t very well suited for the use case (which is basically FaaS-like but with long-lived jobs).
The “split” workflow you described is basically what we were doing (but with AWS Batch instead of Dokku); it’s just that it’s more cost-efficient to consolidate everything into one cluster (especially with preemptible gke nodes) and also better to have a common set of tooling for the Ops team.
To be fair, we haven’t yet completed the move from Batch to k8s so it’s possible that part of the plan won’t pan out as expected.