News for hackers. Stuff that matters.
Re:The sad state of sysadmin in the age of containers (2015) by spiderjerusalem on Wednesday May 16, 2018 @01:37PM (#17084487)
What fascinates me about this is, and sorry for being morbid, but what happens when y'all die? Does knowledge of the lower levels of the stack go away with your generation, or will there be enough of us young ones picking the important stuff up?
Re:The sad state of sysadmin in the age of containers (2015) by Twirrim on Wednesday May 16, 2018 @02:01PM (#17084777)
There's rarely anything old school sysadmins have learned that hasn't come from experience.

Been there, done that, fought that shit the first time. And the second. And the third. (it's amazing how often I find myself solving what are essentially the same problems over and over.) It's one reason why you'll find we'll push back on the "ohh shiny". There are many wonderful and fascinating things coming out. Tech is an amazing field to be working in. But it's also ridiculously frustrating because no one pays any attention to _why_ things are done the way they are, or _why_ approaches haven't worked in the past (I'm all for re-introducing past failed approaches, as long as there's evidence those reasons have been investigated)

You'll find a common trend amongst us in that most of us sort of ended up in the role accidentally. Schools and college teach you to become developers. Few people tend to head to college with the view of specialising in the ops side of things.

Even speaking as a comparatively old-school sysadmin, my strengths come from being flexible and adaptable. What I do today is nothing like what I was doing 5 years ago, and what I did then is nothing like what I was doing 5 years before that, and so on down the line. The field is constantly in flux.

I just have the best part of two decades of experience to both anticipate the problems, and be able to get to diagnosis quicker when things do go wrong.

Even as the older sysadmins die off I'm fairly confident there will be newer ones to replace them, because people are going to continue to learn from the problems they run in to.

Re:The sad state of sysadmin in the age of containers (2015) by ams6110 on Wednesday May 16, 2018 @03:10PM (#17085487)
Ansible is one "ohh shiny" thing that has greatly increased my productivity as a sysadmin. Before that I would automate what I could with ssh and pdsh and scripts, but it was never as well polished as Ansible.

I'm even using ansible for ad-hoc stuff (tweaking a config, restarting a service) because it's easier to do that from a management server than log in to some remote host, get oriented as to the OS distribution and version, and run commends in the shell there.

Re:The sad state of sysadmin in the age of containers (2015) by noir_lord on Wednesday May 16, 2018 @03:34PM (#17085696)
I like ansible with vagrant as well - it makes for a nice clean way of deploying to development environments while also been nicely 'self' documenting and not limiting (you can drop back to shell), it's a lovely tool for the most part.

Edit: The thing I really like about Ansible is how unsexy it is, it's just a nice sane way of doing largely what you could do yourself with ssh and bash but in a language that doesn't make you want to cry.

I've been around linux since the 90's and Ansible feels comfortable, predictable and stable - what you would want in a piece of software that can be mission critical in the most fundamental sense.

Re:The sad state of sysadmin in the age of containers (2015) by dozzie on Wednesday May 16, 2018 @05:51PM (#17086794)
It would be even nicer if people weren't advocating it as a configuration management system, only as what it is: a deployment system.
Re:The sad state of sysadmin in the age of containers (2015) by piva00 on Thursday May 17, 2018 @05:13AM (#17089859)
I think I achieved my perfect balance of tooling for current systems with Ansible and Docker.

Ansible automates even provisioning in AWS. I never really liked CloudFormation's way of creating stacks, so I began to use Ansible to document the application's stack, have used it to deploy systems running in EC2 with RDS, ElastiCache, SQS, SNS, DynamoDB, etc.

After provisioning/configuring I'd end up with an instance in EC2 with Docker installed and from there our CI/CD would just trigger the deployment playbook that simply would do a `docker pull` of the version tagged for release and start the container.

Ansible helped to also install our Splunk forwarders as running it from Docker was a hassle still not so long ago, so we would have the best of both worlds: configurability of the host machine completely with Ansible and packaging and predictability of deployment through Docker.

I advocate this stack as simple enough to learn and use with widely used tools without their fancy (and often broken) features. Even though they can be still a bit immature, they are production-ready enough.

Re:The sad state of sysadmin in the age of containers (2015) by ci5er on Wednesday May 16, 2018 @10:24PM (#17088222)
Who advocates it as a configuration management system? I've not seen that...
Re:The sad state of sysadmin in the age of containers (2015) by resonator on Thursday May 17, 2018 @02:16AM (#17089179)
Re:The sad state of sysadmin in the age of containers (2015) by nunez on Wednesday May 16, 2018 @11:34PM (#17088513)
Everyone compares it to chef and puppet
Re:The sad state of sysadmin in the age of containers (2015) by ci5er on Wednesday May 16, 2018 @11:42PM (#17088544)
Ansible? I could see using it to build a container that then got "orchestrated", I guess, but ... hmmm. I've never really looked at them as doing the same thing. (Nor have I looked at either Chef/Puppet for CM. Maybe I'm just stupid... My ex- certainly thinks that I am...)
Re:The sad state of sysadmin in the age of containers (2015) by switch007 on Thursday May 17, 2018 @04:16AM (#17089657)
It calls itself one, I imagine many people also consider it one. What is a true configuration management system?
Re:The sad state of sysadmin in the age of containers (2015) by Jedd on Thursday May 17, 2018 @04:39AM (#17089756)
I used puppet -- unhappily -- for a while before discovering SaltStack.

Salt, like Ansible, is a response to the Puppet/Chef hegemony -- ruby and DSL's and tacked-together bits that are a nightmare to install and upgrade in themselves).

I'd suggest Salt did some things much cleaner than Ansible, and while it can be an orchestrator, and deployment system, it also excels as a configuration management system ... but I think typically people are talking about configuration management systems that are tightly coupled with more formal change management systems (of which I've found pretty much none that work well).

Re:The sad state of sysadmin in the age of containers (2015) by scurvy on Wednesday May 16, 2018 @02:03PM (#17084810)
It's a legit concern. There was a NANOG panel about this exact thing. I believe the quote was, "Take a look around. We're all old and greying. We have a severe pipeline problem." And then much to AWS' dude's dismay, the topic shifted towards blaming cloud services because no one takes the time to learn how any of this works any more.

Want to guarantee your child's future employment? Don't just teach them to code (the machines will do that). Teach them how to build networks and truly understand network protocols.

Re:The sad state of sysadmin in the age of containers (2015) by setquk on Wednesday May 16, 2018 @02:14PM (#17084938)
I’m going to teach my children how to navigate the world of insane Harry-Potter-esque rules which all IaaS/PaaS platforms enforce upon you. They will become software language lawyers and be masters of the electric Disney dollar.

You know like “ahh don’t call the messaging endpoint more than 800 mega-milli-times per mega-nano-second or it will cost you three bazillion CPU credits, but only on three and a half cores which will starve all your instances, issue an invoice and proceed to melt your credit card.”

Re:The sad state of sysadmin in the age of containers (2015) by w8rbt on Wednesday May 16, 2018 @09:36PM (#17087983)
DDOS is now a billing issue.
Re:The sad state of sysadmin in the age of containers (2015) by kazen44 on Wednesday May 16, 2018 @02:23PM (#17085035)
> Teach them how to build networks and truly understand network protocols.

I don't know how the situation is in the US, but in my country network engineering is actually quite a popular field of study. (we have college level education in network engineering).

The one thing that stands out though is that it's mostly done by youngsters who have either sysadmin experience, or worked in IT before that. Almost everyone who comes from high school goes into Software Engineering.

I think this is mainly because networking is quite an invisible field so to speak. Many people don't even know your job exists, and many young people only see the shiny hip side of it. (being Software Engineering).

Being good at network engineering is hard, especially once you get past entry level work and actually start being responsible for designing large-scale networks. Mainly because building a network is a major financial investment where garuanteeing performance is hard without either a ton of experience, or a shitton of lab time.

this kind of work pays very, very well though.

Re:The sad state of sysadmin in the age of containers (2015) by bordercases on Wednesday May 16, 2018 @04:19PM (#17086095)
What country?
Re:The sad state of sysadmin in the age of containers (2015) by zamber on Thursday May 17, 2018 @09:54AM (#17091230)
Netherlands
Re:The sad state of sysadmin in the age of containers (2015) by rootlocus on Wednesday May 16, 2018 @02:56PM (#17085355)
> Don't just teach them to code (the machines will do that). Teach them how to build networks and truly understand network protocols.

Building networks and understanding protocols is something machines can do TODAY. The entire internet was built to survive a nuclear war. It can reshape itself, and it most definitely understand network protocols. By definition, protocols are the language of machines over the network. And it's been like this for a few decades.

The only reason knowing low level network protocols in a world where machines can code anything (which makes them better analytical thinkers than humans) is to beg the machines for mercy in their ancient tongue.

Re:The sad state of sysadmin in the age of containers (2015) by gaius on Wednesday May 16, 2018 @03:37PM (#17085720)
It can reshape itself, and it most definitely understand network protocols

It really, really can't! That quote about the Internet interpreting censorship as damage and routing around it? Or the one about information "wanting" to be free? Taken wildly out of context.

Re:The sad state of sysadmin in the age of containers (2015) by rootlocus on Wednesday May 16, 2018 @03:46PM (#17085813)
Then what about routing protocols such as RIP, OSPF, etc? I'm sorry, but I don't know what quotes you're referring to.
Re:The sad state of sysadmin in the age of containers (2015) by jauer on Wednesday May 16, 2018 @05:02PM (#17086466)
Routing protocols convey state information between routers, but really that is just table stakes.

So, what do network engineers do? Classically, Set up and troubleshoot those systems. Currently, transforming from manual work to building systems to deploy, monitor, and remediate routers and such. In other words, the same stuff sysadmins->(SRE|PE) folks do and undergoing a similar transition.

SRE methods are quite applicable to neteng: https://landing.google.com/sre/book/chapters/eliminating-toi...

Re:The sad state of sysadmin in the age of containers (2015) by gaius on Wednesday May 16, 2018 @03:52PM (#17085865)
Then what about routing protocols such as RIP, OSPF, etc?

Don't forget BGP. I don't think they do what you think they do, at least, not to the extent you think they do them. There is a hell of a lot of manual work in running any sizeable network even within a single organisation.

Re:The sad state of sysadmin in the age of containers (2015) by rootlocus on Wednesday May 16, 2018 @03:54PM (#17085880)
And what exactly do they do that I don't understand?

> There is a hell of a lot of manual work in running any sizeable network even within a single organisation.

I'm not trying to say they do everything, and no manual work is required. I'm trying to say machines are already doing part of it. OP believes that give a world where machines can code, they can't design or maintain networks, which I find truly ridiculous, since machines are pretty far from doing any kind of "coding" today, but they do networking and network protocols pretty well.

Re:The sad state of sysadmin in the age of containers (2015) by gaius on Wednesday May 16, 2018 @04:07PM (#17085992)
since machines are pretty far from doing any kind of "coding" today

The first attempt at a system to turn plain English that even managers could write into executable code was 1959 - COBOL. So you're right in a sense, even 59 years later - but also wrong if you think networks are any more advanced than this. The Internet really cannot "reshape itself" and probably never will be able to.

Re:The sad state of sysadmin in the age of containers (2015) by spc476 on Wednesday May 16, 2018 @09:42PM (#17088023)
RIP and OSPF are interior routing protocols---that is, they're used for routing within an organization (or autonomous system in Internet lingo) and deal with technical routing issues (fastest link, most bandwidth, etc). BGP is for routing between organizations and deals with political issues than technical issues (we need to send all traffic here due to contracts, unless it goes down, then shift traffic over there; and refuse routing information from such-n-such organization because they don't have their act together).
Re:The sad state of sysadmin in the age of containers (2015) by dbt00 on Wednesday May 16, 2018 @07:02PM (#17087163)
Routing protocols are how computers communicate network admin instructions to each other really really fast.

They're still just tools wielded by humans, even at a distance. Even today.

Re:The sad state of sysadmin in the age of containers (2015) by busterarm on Thursday May 17, 2018 @02:31AM (#17089249)
Moreover, who is going to write the next protocols we need?

This isn't a field that is done inventing. Not even remotely close.

Re:The sad state of sysadmin in the age of containers (2015) by rootlocus on Thursday May 17, 2018 @02:42AM (#17089302)
OP was talking about a future where developers become obsolete because machines take over the development sector. Do you think writing protocols will be something humans will do better than machines?
Re:The sad state of sysadmin in the age of containers (2015) by busterarm on Thursday May 17, 2018 @12:27PM (#17092780)
Yes. Machines might come up with something that's 'good enough' though.
Re:The sad state of sysadmin in the age of containers (2015) by zippitydoodah68 on Wednesday May 16, 2018 @02:43PM (#17085216)
Been working since 86 in this industry.

You lose and gain and you should always be mindful of what is coming. Humility is good. I love you young guys, your ideas keep coming and they are mostly good.

Re:The sad state of sysadmin in the age of containers (2015) by twic on Wednesday May 16, 2018 @03:29PM (#17085651)
Same problem as making sure your system doesn't lose data when a server dies. Make sure you have enough copies of the knowledge by propagating it between people. Try to have some kind of offline recording (books?) for recovery from a disaster where you lose everyone. Have an idea of how to recover at a business level if you do lose the data forever.

The trouble is making sure these plans actually work. This is why Netflix randomly execute some of their employees every month.

Re:The sad state of sysadmin in the age of containers (2015) by jlgaddis on Thursday May 17, 2018 @09:04AM (#17090941)
> This is why Netflix randomly execute some of their employees every month.

I certainly hope this practice doesn't start catching on with other employers.

Re:The sad state of sysadmin in the age of containers (2015) by telchar on Wednesday May 16, 2018 @09:14PM (#17087849)
"This is why Netflix randomly execute some of their employees every month"

Would that be their King Kong application?

Re:The sad state of sysadmin in the age of containers (2015) by dsr_ on Wednesday May 16, 2018 @03:04PM (#17085425)
That's part of the reason why my last two hires have been at the beginning of their career. For both of them, it was their first major sysadmin responsibility after having jobs involving tech support and occasional Linux experience.

The key is to pick smart people who are good at learning and find complex systems interesting. Then, of course, you need to have interesting projects for them to work on.

Re:The sad state of sysadmin in the age of containers (2015) by alekratz on Wednesday May 16, 2018 @04:59PM (#17086441)
It's a very real concern. We have a thing called a "bus plan" for all of our tech employees (6 of us - small non-tech company). It basically attempts to cover everything that we would need to know if one of us gets hit by a bus.
Re:The sad state of sysadmin in the age of containers (2015) by seorphates on Wednesday May 16, 2018 @02:13PM (#17084931)
They'll never die.

Some monkeys will just keep swinging branch to branch. Some monkeys check out the tree.

Re:The sad state of sysadmin in the age of containers (2015) by karlkatzke on Wednesday May 16, 2018 @02:24PM (#17085047)
What makes you think there aren’t young people doing systems administration? Our last two hires in my most recent job were 23 and 27 respectively. Sure, they’re getting trained in the new hot cloud stuff... as the grumpy seniors figure it out first and set patterns... but they are still doing daily work with some rather ancient stuff.
Re:The sad state of sysadmin in the age of containers (2015) by spiderjerusalem on Wednesday May 16, 2018 @03:12PM (#17085506)
I'm not saying there are no young people doing sysadmin. What I'm trying to say is that if the new 'infrastructure' that all sysadmins learn is not an open UNIXy system where you can grok all the internals if you want to, but closed systems owned by 2-3 major cloud players, then we kinda maybe have a problem in 20 years?

Of course, one can argue that that will just cause a new wave of openness and the cycle continues.

Re:The sad state of sysadmin in the age of containers (2015) by tapoxi on Wednesday May 16, 2018 @04:10PM (#17086017)
As a "young" (30) sysadmin/devops dude I think that open, Unixy system is Kubernetes. I can take an application, dockerize it, write a helm chart and run it anywhere.

The risk is in treating anything as a black box, whether its a managed service or a container you pull from Dockerhub. It's something you'll get burned by eventually and need to learn from experience.

Re:The sad state of sysadmin in the age of containers (2015) by geggam on Wednesday May 16, 2018 @07:34PM (#17087314)
You setup and manage K8s and all the kernel tunings for the host systems so the NAT layers are optimized ?
Re:The sad state of sysadmin in the age of containers (2015) by karlkatzke on Wednesday May 16, 2018 @03:52PM (#17085868)
Yeah, and my point was that we start the young people that we hire on the open stuff. Then we move them up and on to the other open stuff, which runs on top of the cloud vendors.

(As almost everyone else points out, the closed cloud vendor stuff is nowhere near flexible enough for most moderately complicated use cases unless you’re running at serious scale.)

This is a false crisis.

Re:The sad state of sysadmin in the age of containers (2015) by swarnie_ on Wednesday May 16, 2018 @03:20PM (#17085569)
My company seem to split responsibilities based on age. 40/50 year olds deal with oracle Linux, AIX and Solaris. Under 30's hires are more focused on cloud and mobile. We're all expected to have a footing in Windows and Oracle DB.
Re:The sad state of sysadmin in the age of containers (2015) by user5994461 on Wednesday May 16, 2018 @01:44PM (#17084573)
Everyone will be running on AWS, Google Cloud and Azure, who know how to operate infra that doesn't crash all the time.