My last team was the same, there was ~100 developers on the main app, ~50 on the ancillary services, and a ballpark of ~10 on the "platform". I was just a team member on that team. Current org is 15 devs, 30 people total.
> My concern is that there is a cost to magic of adding deps like that.
Of course there is, there's such thing as a free lunch. It's up to you to decide which is right for your team/org. The cost of doing it manually is developer time, tech debt and the bus factor. The cost of using helm is introducing an abstraction. Profesionally, the bus factor and developer time of home rolling a solution make relying on existing solutions a very appealing proposition. Having to maintain a set of half baked shell scripts is exactly what tools exist to solve, and I've had more issues with deployments where someone has bunged together a bash script that makes varios assumptions about tooling, versions, installation paths, working directories, path formats, etc. than I have with problems related to our actual deployment infrastructure.
> I guess my question is at what org scale the tradeoffs are right
The cost of switching is huge, and likely not worth it unless you have genuine problems that would be solved by moving. For a greenfield project, frankly I'd think you're wrong to use docker-compose on any new project. Either you're managing the underlying infrastructure (ec2/vps style) or you're not (fargate/DO App Platform style), and in either case docker compose requires roughly the same amount of investment for a smaller return on a less actively developed product objectively solves less problems and requires more work to integrate.
[0] https://github.com/marketplace/actions/deploy-to-kubernetes-...
> Coming from AWS to Linode VPS with Caddy + Docker felt great
I guess that depends what you were using with AWS, but Fargate + k8s is a pretty amazing workflow.
> Im wondering at what point (N requests per second ?) it becomes tech dept
I make these calls for our org, and I wouldn't call it tech debt based on the performance or scale, but because of maintenance. As an example, if you needed to hire a new person/replacement person for your team, would it be easier to find someone who can maintain k8s or docker compose?
The other area it shines for is the ecosystem. If you need to add monitoring to your app (e.g. with Prometheus), adding it to a k8s cluster is `helm install prometheus`