But honestly, how often is a web applications bug due to a kernel bug?
b) In my experience, the application developer is held responsible for the application's behavior in production. In the luckiest .01% of scenarios, there might be an infrastructure engineer with appropriate permissions and free time trolling the Slack support channel at the moment you report the issue. Otherwise 99.99% of the time infrastructure will not acknowledge of investigate anything complicated or subtle with just one service owner complaining. The infrastructure group, organizationally, is graded on shipping new platform features and on coarse KPIs for the performance of the platform as a whole; nobody is getting paid to investigate the weird bugs of some application team somewhere.
No, it doesn't eliminate such distinctions. My view of DevOps is more about ensuring that automation is used as much as possible to meet objectives.
It's definitely not about making everyone a homogeneous developer unit that can work on every problem.
People come in all shapes and sizes, some are more competent with certain things than others, others have a lot more experience with certain things. That's aside from the whole preference thing - not everyone wants to or has an interest in managing infrastructure.
Maybe when you have a handful of developers and a small set of infrastructure, that's fine - but at a certain point you start to require more and more specialised knowledge. Yes, even when you're all-in on Cloud and using all the SAAS/PAAS products out there.
>Otherwise 99.99% of the time infrastructure will not acknowledge of investigate anything complicated or subtle with just one service owner complaining.
Yeah, that's an organisational problem from the sound of it.
I think it’s best to consider the original source which is this talk from Flickr: https://youtu.be/LdOe18KhtT4
("10+ Deploys Per Day: Dev and Ops Cooperation at Flickr").
Directly it talks about joining developers and operations into the same team- later Patrick Debois would refer to this as DevOps and a year later the first DevOps days in Ghent was organised (also by Debois).
We really need some new terminology.
I swear, sysadmins were annoying as an application developer, but devops is something else.
People with one year of actual work experience get hired as devops, and have all the privileges I would need to fix their mistakes, but I can’t, because I’m an ‘application developer’. So instead you end up teaching them how to do their job.
I’m not salty at all.
If your programmers aren't delighted with you, I'd say you the Ops person is not practicing DevOps or you have a buy-in problem to DevOps practices at an organization level.
My previous role had my title as "DevOps Engineer" but it always rubbed me the wrong way. I was just an Operations Engineer with a focus on making my developers' jobs easier, in any way I could. Having that as my North Star kept me honest about the work I was doing versus considering the role more like Operations Engineer v2.0.
In the Silicon Valley, at least, DevOps seemed to be (seems to be?) sort of in vogue; I think it's important to keep its core qualities of bridging Development and Operations in mind as opposed to just shifting an existing position's title in an attempt to attract talent.
And this should extend throughout the organization. If Architecture or Security or any other group is making your life miserable, they too should be DevOps'ing, working closely with you, caring about your frustrations that only they can fix. Sadly there are still so many silos left to break up.
Most people with such a title are actually something like “Automation Engineers”, “Infrastructure Engineers”, “Operations Engineers”, “Site Reliability Engineers”, etc, that are involved in a DevOps “process”, “initiative”, “culture”, etc.
You can write the same screed full of generalizations from the perspective of any job title: a devops person would lament the fresh-out-of-bootcamp “application developers” who have no idea how systems work together so write SQL queries that retrieve a million rows, one at a time. “Works on my local!”
Saying the emperor has no clothes is not white tower thinking,
However it’s also misguided to assume that specialities don’t exist. You can have infrastructure guys, developers, security folk; and there will be overlap between each role but it’s impossible to be the master of each trade.
I agree that arrogance is an unpleasant trait but arrogance can take many forms, rudeness to colleagues, or over assuming ones own technical capabilities in adjacent fields.
I'm being a little flippant here though, I am aware that developers are often asked to wear the sysadmin hat too.
Giving everyone production SSH experience is, in my experience, a way to run into all sorts of weirdness, not to mention endless frustration.
In a modern automated infrastructure, that box is likely a container running on a virtual machine that's ephemeral and can (and probably will) go away at any moment based on any number reasons - maybe CD kicked off a new deployment, or maybe the load changed and the instance was selected for scale-down, or maybe our spot bid for that AZ isn't sufficient for keeping the instance around, maybe you being SSHed in and poking around impacted the health-check, and so it's being killed for not performing right.
Theres many other problems, too - lots of applications are built in some way that there's simply no other way than secrets (passwords, api tokens, keys) to reach other systems, particularly third party systems. So production boxes have production secrets, which you probably don't want to share with everyone.
Giving everyone SSH access so they can, in theory, take nginx/kernel dumps as needed tends to imply giving superuser rights, which means they can do whatever they like.
So, yes, pull in someone else - find some way to try and reproduce the problem NOT on production, if that fails, perhaps there's a way to grab enough detail or pull additional logs or network captures to identify the issue. If that fails, well okay, lets SSH in - but we need to coordinate that to ensure that instance does't go away, and doesn't impact production while you do it.
There is also the case of ratios. An organization probably needs more developers in specific areas than DevOps, so with dedicated DevOps you could concentrate similar work from across several teams to a dedicated DevOps team that knows that work very well.
You can't hire twenty developers that all have the same skill/inclinations, the same interests, the same experience.
That's not to say that a DevOps Engineer is some super 10x rockstar developer - no, they're going to have the same variations on skill, interests, experience, etc.
It depends on your environment, but there's so much different tech once you count the entire stack, that I don't think it's reasonable to expect any one person to be an expert on all of it, or even a lot of it.
> > Every developer needs access to some servers for example to check the application logs.
> I fundamentally disagree with this.
So,
Developers shouldn't be reaching for SSH access to check logs.
If you're encountering problems that you can't diagnose through the existing logs, then you should probably be involving at least one other person - someone who has that production experience, who might have some additional knowledge about the problem.
If, and only if you've exhausted other avenues - then reach out for SSH access. But it should be a last resort, not the first resort. Plus, anyone SSHing into production boxes should really be very familiar with how production is configured. You can do more harm than good by poking around on a production box being completely unaware that you're causing alarms and outages elsewhere because you taking a memory dump of nginx caused in-flight requests to get timeouts and so-forth. The people with that experience are generally the DevOps/Infrastructrue folks since they're the ones who deal with production all day, and are going to get the pages if something goes wrong with that.
Yeah, perhaps. It'll depend on the circumstances, right.
I've got a reasonable amount of GSuite and Exchange experience, and same for Active Directory. I'm reasonably confident that I can work my way around those and do most of what I need to do without breaking it.
I needed some GSuite groups set up, some folks added to them, and a GSuite OAuth application set up and some values passed back and forth to do some integration. The only way to do all of that is with full GSuite Administrator rights.
Now, I could ask why they don't just give all Devops folks GSuite admin rights, it'd be much easier (for me) and I could do my job more efficiently.
The response is going to be something along the lines of:
> You don't need that access most of the time. > The times you do need that access it's often for a limited time or scope, and to resolve a specific problem. > For now, It's better that you work with someone who is responsible for that stuff on a day to day basis to do those things.
This is, in my opinion, pretty reasonable. Sure, it meant more delay until someone was available to do the GSuite configuration, and we needed to jump on a call to pass IDs back and forth and test it out. But it got done, and it wasn't overly burdonsome.
They didn't hire me as a GSuite, Exchange or AD Admin - they already have folks to handle that. That I have that knowledge and experience is still useful for the company - I know exactly what to request is done, and we can talk on the same level about it. Heck, if it turns out that I need this every day and I'm constantly going back and forth with them on setting things up - then I might get that access, but it'll probably come with an explicit requirement that I use it in specific ways, that I follow their processes/procedures, and keep them informed on when I'm using it and why
That is the responsibility of system administrators. Application developers have no business on a production machine. If your sysadmins don't have the technical skills to diagnose these problems, they are incompetent and must be replaced.
The actual result of this is that the sysadmins are not replaced, and the application developers end up in an emergency conference call at 3am to tell the sysadmins which buttons to click on the production environment, since they’re not allowed access themselves.
DevOps is not a role or role segregation, it’s about aligning incentives and outcomes across functions in an org (hopefully through collaboration, tooling, and knowledge transfer).
The caveat is that if your org is fundamentally broken, none of the above applies or works and it’s all lipstick on a pig.
> ship log events out of the box and into something searchable, indexable, and can derive metrics
That's just a way to spend a ton of money. There's really not a reason to ship or index logs. Just leave them where they are produced and instead of shipping them to a central thing that inevitably becomes an expensive bottleneck, instead develop ways to access them in place (other than SSH) and frameworks for large-scale distributed searching where you push the predicate out to the machines where the logs exist.
This is the age old problem of simple problems being harder to solve at scale. Running a blog on Digital Ocean is very different to running a million dollar application on AWS.
[1] https://medium.com/@Joachim8675309/devops-concepts-pets-vs-c...
I'd rather not engage in pointless unprovable arguments, if you don't mind.
> The problem is that you are treating your logs as pets when the truth is the logs are also cattle. Virtually all debug logs will pass through their life cycle without being read, so indexing them is just a flagrant waste of energy.
You shouldn't have debugging enabled on production systems unless you have an interim process that filters out debug messages before they get indexed (and thus you can toggle which logs get indexed there rather than reconfiguring / redeploying all of your application nodes).
Also nobody is suggesting logs should be treated as "pets". You still want to purge out older logs however the problem is you cannot always replicate reported errors so if you don't have those log messages captured then you're sod out of luck.
Don't get me wrong, there is a certain allure to the traditional method of systems administration - I've been on both sides of the fence - but central logging services have so many other benefits such as security (tamper proof logs, users don't require SSH), ease of use, persistent logging, etc. The only real downside is cost but that quickly becomes absorbed in your pricing plan when customers start asking for SLAs.
By the time you want centralized logs, you are probably already spending much more than the logging will cost on infrastructure.
What you're doing here is constructing a straw man argument while agreeing with the same point I was making.
> > ship log events out of the box and into something searchable, indexable, and can derive metrics
> That's just a way to spend a ton of money. There's really not a reason to ship or index logs...
That's what I said. ;)
> One of the ways to make it seem a reasonable suggestion is the broader topic of not logging into machines at all, ever, and why that may be a good thing.
Except none of that was being discussed. Only log access.
> They're explicitly talking about not logging into machines for log access.
Why can't you just admit that you didn't read the thread properly rather than insulting all of our intelligence with these piss poor mental gymnastics where you redefine the context of what people had very clearly written.
The conversation you're replying to, however, did. I know this for a fact because I was involved in that conversation and I made that distinction myself ;)
I guarantee that you haven't, as the number of people that can make that claim in relation to my background is vanishingly small and I know almost all of them by name.
I centralize logs. Not only does it make more sense for administration at scale, it's invaluable for security reasons and assists in compliance by providing a controllable guaranteed audit trail.
You may have a valid argument about cost here for some applications, but it's unwise to make arrogant claims you cannot back up.
Not only is that vanishingly unlikely to be true with this crowd it’s also not at all the kind of rhetoric that they’ll listen to either.
Your storage/indexing money spend scales with your company. If you have < 10 hosts you can probably handle logs with a cost that's a rounding error with something like loki+grafana.
If you have a bigger environment and more people, the cost will be more significant, but you'll also likely have more revenue to justify it.
Proper Information Security policy is one very good reason. If a box is compromised, you don't want the logs being tampered with.
You ship the logs to a central, dedicated server so that a clear audit trail can be established.
On the point, what do you do if the machine is not there anymore? Your model fails immediately. Also, some processes operate on an assumption that the "machine" will stand-up -> do-something()- > die(), which your model also doesn't support. Not to be messing with you, but I don't think there is the "right" model, but whatever works for us. I do like your model for immediate accessibility of the logs.
Anyone who needs access could remotely run a command that would return the results of whatever permitted query they submitted.
Actually, in my company, we are pushing the logs to an external service, so I guess that line I wrote there was not totally thought out. ;)
However, it still might be useful for certain people to check servers for health or other metrics and you might not always have everything as perfect as you wish. I guess you already stated that yourself.
So in general I'd agree. Good comment!
Also, if you aren’t on “cloud”, odds are you are still using something like Kubernetes or DC/OS or Swarm or the like. If you aren’t then well, wordpress sites aren’t really in the same ballpark technically. (Joke, Wordpress sites get traffic, some lots of traffic, I don’t discriminate against the PHP tribe).
However, for anything shy of full serverless, just s/developer/ops and you still have to solve for the same problem. At the end of the day you're always going to want someone to be able to get a shell on the box for troubleshooting some critical issue. If there is a box.
Great! hand waving. What are you going to do is this log streaming process stops working, and cannot restart? Kill the machine and lose the logs? That might be ok for your webapp, it could be trouble for more serious applications.
> You’d be surprised at how much less stress there is when you can have alerts on log events from your application when things break instead of a support call or a support ticket.
Giving SSH access doesn't mean you don't have event logs.
> stopped teaching people to treat their app and server as a second home, as a pet, that must be nurtured
In our case, nobody is treating anything as a pet. Our servers are mostly spot/preemptibles, but we still have SSH access. Because it makes it really easy to debug if things do go badly wrong. From being able to see htop output, to being able to upload custom binaries to investigate an issue, or to smoke test a code change... it's a huge time saver.
Give me SSH or give me death.
This kind of access shouldn’t be your go-to but it’s important to have a battleshort.
But I would be lying if the typical logging setup was this simple. It’s basically all remixes of ELK and while they’re fantastic when they work there are a lot many moving parts that can individually fail.
I fundamentally disagree with this. I’ve been writing software a long time and I used to demand I have server access so I can tail logs, creating the problem this article talks about resolving (good read btw). But I can’t help but wonder why we keep teaching this mindset. We have log analysis tools available on pretty much every cloud now. Docker has an aws log driver, a gcp log driver... etc. Backend developers should make a conscious decision on how to ship log events out of the box and into something searchable, indexable, and can derive metrics from. I ran a cloud infrastructure group that vehemently said “No” to any dev requesting ssh access. Not because they couldn’t, but because we don’t have access either. We created our platform without the ability to modify it. Infrastructure as code. Only way is to redeploy the stack. You’d be surprised at how much less stress there is when you can have alerts on log events from your application when things break instead of a support call or a support ticket. Proactive > Reactive debugging. I also understand not all shops are at that level of maturity. I’d love it if the community as a whole stopped teaching people to treat their app and server as a second home, as a pet, that must be nurtured. Obviously these are my opinions and the article itself addresses how to handle ssh keys for server access in a logical way, my only issue is why create that mess in the first place?