News for hackers. Stuff that matters.
Re:As AWS Use Soars, Companies Surprised by Cloud Bills by Tistel on Monday February 25, 2019 @12:53PM (#19247647)
I have worked at a startup that was paying > 20k per month for AWS. I am certain the whole system could run on one dedicated blade. There was just this culture (made easy with docker containers) of "spin up another box" for every silly little tool. I have met self taught dev ops people who did not understand that you could create more than one user on a aws ubuntu machine. I was like: "if cost is a big issue, then lets spin up one beefy machine and create users for each dev", because thats what everyone using unix did back in the day (it works fine). The reply was "thats not possible" so every dev got their own dedicated instance that was 99% idle. We are paying >3k per month to aws and the business team is complaining about waste. sigh. old guy rant over.
Re:As AWS Use Soars, Companies Surprised by Cloud Bills by thruhiker on Monday February 25, 2019 @01:25PM (#19248002)
Did you weigh these costs against the value of the developers not stepping on eachother's toes using the same instance?Separate users with sudo privileges provides zero isolation.

I'm unsure what use case was, but if it was a testing or staging environment dedicated to each developer, having a separate right-sized instance for each is quite sane and reasonable.

EC2 instances such as the T2/T3 series are cheap enough that there's normally not a need to be so miserly in my experience.

Re:As AWS Use Soars, Companies Surprised by Cloud Bills by CharlesColeman on Monday February 25, 2019 @01:30PM (#19248050)
>> I am certain the whole system could run on one dedicated blade. There was just this culture (made easy with docker containers) of "spin up another box" for every silly little tool.

> Did you weigh these costs against the value of the developers not stepping on each other's toes using the same instance? Separate users with sudo privileges provides zero isolation.

Each developer's machine is a separate instance. If the whole system could run on "one dedicated blade," then the developers could easily run a local instance on their own machines (either directly or in a VM) without stepping on anyone's toes.

Using AWS for local development seems wasteful and over-complicated to me. The only good reason I can see for doing it is if you're really vendor-locked into AWS-proprietary services, and it's impractical to run your code anywhere except on AWS (but that's arguably a mistake in itself).

Re:As AWS Use Soars, Companies Surprised by Cloud Bills by deathanatos on Monday February 25, 2019 @02:12PM (#19248430)
> Each developer's machine is a separate instance. If the whole system could run on "one dedicated blade," then the developers could easily run a local instance on their own machines (either directly or in a VM) without stepping on anyone's toes.

What the parent you're responding to likely meant was that, by running everything on one system ("one dedicated blade") you are causing all of the things on that system to share certain things. The set of system-installed packages; if nginx needs to be installed and running, is its configuration now shared between all of the services on that machine; and even the very OS itself (e.g., if it needs to go down for updates, or it needs to do, say a dist-upgrade, which means that all of the associates services on that instance need to dist-upgrade together, and one laggard can thus hold everyone back.)

Even in the case of developers using an instance on AWS, the same applies, really: a dev, legitimately developing some component, could have cross-talk to others, in all of the same ways listed above. I do agree that giving devs dedicated instances might not be appropriate in many instances, and it perhaps wasn't appropriate in yours. But there's two points here: whether the devs needed those instances, or could develop on local laptops, and whether putting all instances or users on a single instance works well.

Even for development, given that most corps seem to love giving devs that code primarily for Linux servers OS X laptops, there might be advantages to allowing them to develop on the actual target hardware.

> There was just this culture (made easy with docker containers) of "spin up another box" for every silly little tool.

Docker here should make it rather trivial to combine everything onto a single instance, since that's what isolation gets you: a well defined unit that doesn't have endless unknown dependencies on the host system.

Re:As AWS Use Soars, Companies Surprised by Cloud Bills by h1d on Monday February 25, 2019 @07:07PM (#19250758)
Sounds cheaper to just give them VMware fusion and a dev OS image which is mostly identical to the target architecture.

I really don't see why many run their environment on macOS through Homebrew etc which is surely going to require modifications here and there when deploying to Linux servers.

Re:As AWS Use Soars, Companies Surprised by Cloud Bills by miranda_rights on Tuesday February 26, 2019 @01:44AM (#19252456)
An old job had that setup, with about 15 devs per instance. It worked surprisingly well, where each developer had sudo access but was asked to 'use it with care'. It helped that there was a mix in the type of work, so not everyone was doing resource-intensive work, etc.
Re:As AWS Use Soars, Companies Surprised by Cloud Bills by h1d on Monday February 25, 2019 @07:03PM (#19250729)
Can you not run a docker container for each dev who would log into it, so they're isolated to begin with?

Of course ports can't be shared for each app servers but you can get around that anyhow.

Re:As AWS Use Soars, Companies Surprised by Cloud Bills by thruhiker on Tuesday February 26, 2019 @02:10AM (#19252536)
If the development environment is comprised of multiple container images that would likely justify dedicated small EC2 instances.
Re:As AWS Use Soars, Companies Surprised by Cloud Bills by eftpotrm on Monday February 25, 2019 @01:35PM (#19248090)
Knew a place that turned off the developers' test environment because it was costing 10x that per month. Which was as popular and successful as you can imagine. I'm.... skeptical about the benefits of the cloud architecture they actually had, given those costs.
Re:As AWS Use Soars, Companies Surprised by Cloud Bills by twh270 on Monday February 25, 2019 @07:07PM (#19250761)
Our org just implemented this. A script turns off all services in lower environments (dev, qa) at night; another one turns 'em on in the morning. Of course if you need to, you can run your own script to turn on your services "if you need to".

This is being sold as a "step in the right direction", where "right direction" is anything that reduces cloud costs. (Payroll costs are apparently invisible, so the time people waste dealing with this baloney doesn't count. Not to mention the annoyance.)

I'm tempted to point out that they could save a lot more money by merging microservices into macroservices....

Re:As AWS Use Soars, Companies Surprised by Cloud Bills by sigfubar on Monday February 25, 2019 @01:04PM (#19247780)
I've worked for a certain IoT startup that at one point paid $850k/month (eight hundred and fifty thousand USD per month) for AWS usage. After about a year of full time work by a team of SREs and programmers, the bill was made to go down by half.
Re:As AWS Use Soars, Companies Surprised by Cloud Bills by h1d on Monday February 25, 2019 @07:09PM (#19250777)
Any breakdowns on cost?
Re:As AWS Use Soars, Companies Surprised by Cloud Bills by UncleMeat on Monday February 25, 2019 @05:32PM (#19250120)
That's like half of one engineer. Less if you include all of the overhead of having an employee. That can easily be the right choice.
Re:As AWS Use Soars, Companies Surprised by Cloud Bills by mancerayder on Monday February 25, 2019 @01:16PM (#19247910)
I have met self taught dev ops people who did not understand that you could create more than one user on a aws ubuntu machine.

Wait, you're saying there's a Bachelor of Science DevOps degree?

Re:As AWS Use Soars, Companies Surprised by Cloud Bills by Tistel on Monday February 25, 2019 @02:33PM (#19248622)
They are very young, nice kids, who have started from a pure aws standpoint. They started with AWS tutorials. These tutorials did not teach the full unix environment. The gaps in their knowledge is like the grande canyon. If they just read the O'reilly book on linux: http://shop.oreilly.com/product/9780596154493.do. they would be miles ahead. But, their only interest is docker/kubernetes. Those technologies are great, but, they simply have massive gaps in their knowledge of the fundementals. They don't seem to like reading technical books from cover to cover. So they just read tutorials far enough to get the system started. Then they cut/paste from stackoverflow when problems arise. If that does not work, they blame the tools ("couched is too slow!"). Again, its not their fault, just the new dev culture. I freely admit I am an old fogey.
Re:As AWS Use Soars, Companies Surprised by Cloud Bills by mancerayder on Monday February 25, 2019 @04:31PM (#19249641)
We agree.

I interviewed at (a fast growing real estate startup) and as an old guy, I highlighted my vast enterprise Linux engineering experience. The response from management was, oh, that's nice but doesn't matter here. I can teach people the command line in one day. This is a big AWS and exclusively Linux shop.

What do you say to that?

Re:As AWS Use Soars, Companies Surprised by Cloud Bills by heydonovan on Monday February 25, 2019 @04:40PM (#19249709)
From my experience, the hardcore Unix folks should stick to jobs that highlights that type of knowledge. If you join an AWS shop, you're never going to dig into low level things like the proc filesystem, and the arcane knowledge of knowing every single tool just isn't used. I've gone from remoting into machines on the daily, to not using SSH for almost a year now. Just depends on the job I suppose. Went from having knowledge of all the different signals, and getting really good at tcpdump/gdb, to just writing Terraform/CloudFormation scripts, and nuking an instance from orbit if there were any issues with it. Such a different workflow.
Re:As AWS Use Soars, Companies Surprised by Cloud Bills by mancerayder on Monday February 25, 2019 @07:27PM (#19250902)
Yes, BUT:

Grey-haired Linux folks who've hammered on complex, changing, sometimes esoteric systems in different companies with unique challenges COMBINED with having to have learned a programming language, to view these people as obsolete is an ultimate folly of youth here in IT.

To suggest that it's all different now, your Shell/Perl/Webserver/database/caching/networking experience somehow has nothing on Python/Terraform/AWS/Docker really shows inexperience of hiring managers.

I learned both, and I can tell you AWS+someyaml+Python are MUCH easier than any of the stuff I listed above. Yet, there's a new generation of hiring managers that believes otherwise.

Today at work I had to explain absolutely basic DNS to someone. I dunno, call me skeptical about this new paradigm business. The paradigms look awfully familiar.

Re:As AWS Use Soars, Companies Surprised by Cloud Bills by anongraddebt on Monday February 25, 2019 @05:53PM (#19250276)
I'm on the business side, and am relatively young. Have dabbled with computer stuff since middle school (built desktops with my Dad, internet privileges revoked for a year in middle school for hacking the Oracle system, have run linux on my non-work laptop for the past ~5 years, etc.), but never pursued serious study whether solo or in a formal setting.

Both with tech and non-tech related learning, my autodidact process (largely determined by growing up through the late 90's and early aughts) was the following:

(1) Google search or Amazon search

(2) Blitz-read some website I thought might be relevant from the Google search results or Blitz-read for author and subject matter info. I thought might be relevant from the Amazon search results

(3) Stop at (2) because I had pieced together a solution or had gleaned a sufficient amount of information; or repeat (1) and (2) until piecing together a solution or gleaning...

This rather clunky but speedy (notice I did not say efficient) process likely accounts for far more of my knowledge and skill base then I'd care to admit to friends and colleagues.

All this to say: management's response to you is representative of a default mode of thinking determined (to a non-trivial degree) by the environment in which they grew up.

This default mode of thinking has a number of short-term benefits, but often results in many long-term headaches. Indeed, you really can teach someone the command line in a day. However, this someone will end up doing stuff I do with the command line, such as using ls 100x more than is necessary or almost always changing directories one at a time.

Re:As AWS Use Soars, Companies Surprised by Cloud Bills by xfitm3 on Monday February 25, 2019 @07:22PM (#19250862)
“Thanks for your time. Have a nice day.”
Re:As AWS Use Soars, Companies Surprised by Cloud Bills by toomuchtodo on Monday February 25, 2019 @01:19PM (#19247930)
Documentation is no replacement for experience.

EDIT: I agree that OPs choice of words makes the idea expressed ambiguous.

Re:As AWS Use Soars, Companies Surprised by Cloud Bills by mancerayder on Monday February 25, 2019 @01:26PM (#19248020)
I just don't understand what a "self-taught" DevOps person is -- as opposed to a someone-taught? Self-taught usually means "not educated formally."

All DevOps / SysAdmins / etc. are "self-taught." So it's unclear what the OP meant.

Re:As AWS Use Soars, Companies Surprised by Cloud Bills by nunez on Monday February 25, 2019 @11:45PM (#19252037)
it can but if that blade or the enclosure dies, the system is sol...also, procurement can be a pain