News for hackers. Stuff that matters.
Re:Building your own deep learning computer is 10x cheaper t... by mlthoughts2018 on Tuesday September 25, 2018 @07:05AM (#18065226)
In most companies, AWS just becomes a new front-end to the same old IT bureaucracy, and dev teams are still disallowed from creating their own instances or EMR clusters or setting up new products like their own Redshift instance or ECR deployment solution.
Re:Building your own deep learning computer is 10x cheaper t... by ethbro on Tuesday September 25, 2018 @08:41AM (#18065801)
Respectfully, those companies' cloud architects suck.

If someone goes to the trouble to migrate onto cloud, and then replicates pre-devops workflows... wow.

Re:Building your own deep learning computer is 10x cheaper t... by supernovae on Tuesday September 25, 2018 @09:46AM (#18066245)
Not every company is a single page webapp and simple service portfolio... I work at a place with 3,000+ developers and over 700 applications - there is no way in hell our cloud portfolio would have any standards if we didn't have a robust operations/engineering team making it work. Sometimes, even when you adopt the cloud, you realize that your operations model is even more important and there is nothing wrong with that.

The real problem is believing that there is a pure model that works for everyone.

Re:Building your own deep learning computer is 10x cheaper t... by whatshisface on Tuesday September 25, 2018 @10:06AM (#18066383)
The parent's point was that if the ops/deployment engineering team is unresponsive to the needs of developers, it may end up being better to run with no standards in cowboy-mode. If they ops team is extremely fast and highly skilled, they will be a boon full stop. If they are unskilled and politically obstructive, they will be a curse, full stop.
Re:Building your own deep learning computer is 10x cheaper t... by supernovae on Tuesday September 25, 2018 @10:35AM (#18066624)
I got a more "you're doing it wrong" vibe...
Re:Building your own deep learning computer is 10x cheaper t... by ethbro on Tuesday September 25, 2018 @02:33PM (#18069121)
I work at a Top25 US company by revenue.

In no way am I responsible for our migration going well. But I do feel if a company this size can do it, then others failing to do so says more about their architecture teams than the endeavor's futility.

Re:Building your own deep learning computer is 10x cheaper t... by askmike on Tuesday September 25, 2018 @08:52AM (#18065887)
This has less to do with cloud vs own hardware and more about how the company is structured. I've worked in companies before with an ops department: a few people responsible for managing all the cloud servers. All the devs (like me at the time) work locally and have access to an isolated dev/uat environment provided by that team. That team had most of the show automated, they weren't really provisioning any machines by hand.

If something bad slipped into prod there was a process in place of how a dev would work together with someone from that team to fix it.

> pre-devops workflows

I guess this is what you are calling a pre-devops workflow? In a lot of fields not all devs are allowed to see/touch the complete production environment. Not everyone can go the netflix way of "everyone pushes to production and we'll just fix it when it breaks".

Re:Building your own deep learning computer is 10x cheaper t... by tetha on Tuesday September 25, 2018 @12:25PM (#18067760)
I dunno, that sounds like the a good way to do it.

I kinda shake my head if people call for devs to handle all the infrastructure and everything in prod. Why should a developer concern himself with details of scaling and tuning postgres, elasticsearch or loadbalancers? If you don't outsource that, that's the job for the ops team. However, if that's the responsibility of the ops team, there's no reason for dev to have access to these systems beyond the application layer. That just seems like a good way to split the work of running and scaling an application.

Now if we're talking about applications, that's something different. In fact, it's the contrary. I am a big proponent of having developers manage and configure their own applications, including prod. It's fast and makes them develop better applications. However, doing this for critical systems in a responsible way requires automation to enforce best practices. At our place, devs don't have root access to productive application servers, but devs have permissions to configure and use the automation controlling the productive application servers. And it's safe and rather risk-free because the automation does the right thing.

And it's also a different thing if we're talking about test. By all means, deploy a 3 container database cluster to poke around with it. I like experimentation and volatility in test setups and PoCs. Sometimes its just faster to solve the problem with 3 candidate technologies and go from there. Just don't expect things to go into production just like that. We'll have to think about availability, scaling and automating that properly first.

Re:Building your own deep learning computer is 10x cheaper t... by mlthoughts2018 on Tuesday September 25, 2018 @01:12PM (#18068266)
I’m a machine learning engineer. I’d love it if I only needed to worry about machine learning and application interfaces.

Instead, because ops / infra arbitrarily block me from what I need, I have to be an expert on database internals, network bottlenecks, app security topics, deployment, containers, CI tools, etc. etc., both so that I can “do it myself” when ops e.g. refuses to acknowledge some assumption-breaking GPU architecture we need, and so that I can exhaustively deal with every single arch / ops debate or bureaucratic hurdle that comes up for me to endlessly justify everything I need to do far beyond any reasonable standard.

For me, managing devops shit myself is a necessary evil. Far better than the case of unresponsive / bureaucratic ops teams, but worse than the unicorn case of an actual customer service oriented ops team that actually cares rather than engages in convenience-for-themselves optimization at every turn.

Re:Building your own deep learning computer is 10x cheaper t... by tetha on Tuesday September 25, 2018 @02:16PM (#18068962)
This is so frustrating to read as a service provider.

Especially because a well-done infrastructure scales so much harder than the manual style. We're currently dealing with a bit of fallout from a bureaucratic system at one point. People are so confused because to get a custom container build, all they do is create a repository with a specific name and a templated Jenkinsfile and 10 - 15 minutes later they get a mail with deployment instructions. It's so easy, and no one has to do anything else.

Re:Building your own deep learning computer is 10x cheaper t... by meddlepal on Tuesday September 25, 2018 @10:06AM (#18066375)
> I guess this is what you are calling a pre-devops workflow? In a lot of fields not all devs are allowed to see/touch the complete production environment. Not everyone can go the netflix way of "everyone pushes to production and we'll just fix it when it breaks"

Unwillingness shouldn't be confused with ability. Most companies can do this if they're not handling PII/PHI. It takes investment in smart people and time but most companies besides pure software companies see tech as a cost center and avoid investing in better infrastructure and platform systems.

Re:Building your own deep learning computer is 10x cheaper t... by walshemj on Tuesday September 25, 2018 @03:52PM (#18069903)
Very much so when I was arguably doing devops for BT using PR1ME super minis' back in the day. A mate who was in operations for an IBM shop was horrified that our team were allowed to write our own JCL.
Re:Building your own deep learning computer is 10x cheaper t... by ethbro on Tuesday September 25, 2018 @04:01PM (#18069973)
One of my pet peeves is that there's no mandatory "History of ideas in software development" course in most CS curriculums.
Re:Building your own deep learning computer is 10x cheaper t... by solatic on Tuesday September 25, 2018 @09:28AM (#18066126)
A lot of those companies don't have cloud architects, just Amazon/Microsoft/Google sales reps talking into the ears of MBAs about turning capex into opex.
Re:Building your own deep learning computer is 10x cheaper t... by badpun on Wednesday September 26, 2018 @10:33AM (#18076311)
In my current company (a large bank), our team (~50 people) is trying to leave the on-premise infrastructure and move to the cloud because the on-perm stuff is managed in such a way that it's hard for us to acomplish anything. It will probably cost a lot more to use the cloud, but we're gladly willing to pay for it if we can shed the bureucracy this way.

I've read that in medieval times, kings have sometimes abandoned their castles for new ones after too much fecies were accumulated in them (as people were shitting wherever back then). I feel a strong parallel in this story to our situation.

Re:Building your own deep learning computer is 10x cheaper t... by throwaway2048 on Tuesday September 25, 2018 @10:23AM (#18066532)
why not cut out the middle man and just say those companies suck, and the cloud isn't going to fix it for you.
Re:Building your own deep learning computer is 10x cheaper t... by ethbro on Tuesday September 25, 2018 @03:59PM (#18069953)
Because that seems overly reductive.

There are things you can do in cloud (especially as an SMB) that weren't possible on-prem.

To look at those new opportunities and say "No, how can we do things exactly the way we were?" seems like the real mistake.