News for hackers. Stuff that matters.
Re:Building your own deep learning computer is 10x cheaper t... by askmike on Tuesday September 25, 2018 @08:52AM (#18065887)
This has less to do with cloud vs own hardware and more about how the company is structured. I've worked in companies before with an ops department: a few people responsible for managing all the cloud servers. All the devs (like me at the time) work locally and have access to an isolated dev/uat environment provided by that team. That team had most of the show automated, they weren't really provisioning any machines by hand.

If something bad slipped into prod there was a process in place of how a dev would work together with someone from that team to fix it.

> pre-devops workflows

I guess this is what you are calling a pre-devops workflow? In a lot of fields not all devs are allowed to see/touch the complete production environment. Not everyone can go the netflix way of "everyone pushes to production and we'll just fix it when it breaks".

Re:Building your own deep learning computer is 10x cheaper t... by tetha on Tuesday September 25, 2018 @12:25PM (#18067760)
I dunno, that sounds like the a good way to do it.

I kinda shake my head if people call for devs to handle all the infrastructure and everything in prod. Why should a developer concern himself with details of scaling and tuning postgres, elasticsearch or loadbalancers? If you don't outsource that, that's the job for the ops team. However, if that's the responsibility of the ops team, there's no reason for dev to have access to these systems beyond the application layer. That just seems like a good way to split the work of running and scaling an application.

Now if we're talking about applications, that's something different. In fact, it's the contrary. I am a big proponent of having developers manage and configure their own applications, including prod. It's fast and makes them develop better applications. However, doing this for critical systems in a responsible way requires automation to enforce best practices. At our place, devs don't have root access to productive application servers, but devs have permissions to configure and use the automation controlling the productive application servers. And it's safe and rather risk-free because the automation does the right thing.

And it's also a different thing if we're talking about test. By all means, deploy a 3 container database cluster to poke around with it. I like experimentation and volatility in test setups and PoCs. Sometimes its just faster to solve the problem with 3 candidate technologies and go from there. Just don't expect things to go into production just like that. We'll have to think about availability, scaling and automating that properly first.

Re:Building your own deep learning computer is 10x cheaper t... by mlthoughts2018 on Tuesday September 25, 2018 @01:12PM (#18068266)
I’m a machine learning engineer. I’d love it if I only needed to worry about machine learning and application interfaces.

Instead, because ops / infra arbitrarily block me from what I need, I have to be an expert on database internals, network bottlenecks, app security topics, deployment, containers, CI tools, etc. etc., both so that I can “do it myself” when ops e.g. refuses to acknowledge some assumption-breaking GPU architecture we need, and so that I can exhaustively deal with every single arch / ops debate or bureaucratic hurdle that comes up for me to endlessly justify everything I need to do far beyond any reasonable standard.

For me, managing devops shit myself is a necessary evil. Far better than the case of unresponsive / bureaucratic ops teams, but worse than the unicorn case of an actual customer service oriented ops team that actually cares rather than engages in convenience-for-themselves optimization at every turn.

Re:Building your own deep learning computer is 10x cheaper t... by tetha on Tuesday September 25, 2018 @02:16PM (#18068962)
This is so frustrating to read as a service provider.

Especially because a well-done infrastructure scales so much harder than the manual style. We're currently dealing with a bit of fallout from a bureaucratic system at one point. People are so confused because to get a custom container build, all they do is create a repository with a specific name and a templated Jenkinsfile and 10 - 15 minutes later they get a mail with deployment instructions. It's so easy, and no one has to do anything else.

Re:Building your own deep learning computer is 10x cheaper t... by meddlepal on Tuesday September 25, 2018 @10:06AM (#18066375)
> I guess this is what you are calling a pre-devops workflow? In a lot of fields not all devs are allowed to see/touch the complete production environment. Not everyone can go the netflix way of "everyone pushes to production and we'll just fix it when it breaks"

Unwillingness shouldn't be confused with ability. Most companies can do this if they're not handling PII/PHI. It takes investment in smart people and time but most companies besides pure software companies see tech as a cost center and avoid investing in better infrastructure and platform systems.

Re:Building your own deep learning computer is 10x cheaper t... by walshemj on Tuesday September 25, 2018 @03:52PM (#18069903)
Very much so when I was arguably doing devops for BT using PR1ME super minis' back in the day. A mate who was in operations for an IBM shop was horrified that our team were allowed to write our own JCL.
Re:Building your own deep learning computer is 10x cheaper t... by ethbro on Tuesday September 25, 2018 @04:01PM (#18069973)
One of my pet peeves is that there's no mandatory "History of ideas in software development" course in most CS curriculums.