Who would trust a doctor who proclaims his expertise in terms of brand names: "I've got ten years of experience in Cipro and Amoxil. Lately I've been prescribing Zithromax too! It's exciting to write that big Z. I really like Pfizer's community, their Slack is so helpful."

On the other hand, someone who already worked with 10 Languages, 7 webframeworks and 5 deployment/config management tools plus showing good craftmanship has enough experience to learn one more of these quickly. Only for a short term gig, the learning/being productive relation could be disproportional.

I‘ve experienced it being habdled rather the latter than the former way and don’t remember not getting a job for having not yet worked with one of these tools, but bei g chosen for experience and quality work.

Whether we like it or not, software engineering is becoming a trade. Trades aren't inherently low skill or high skill.

What you say about trades is true, but then we need to start making the distinction between software engineers and software technicians. I doubt that will go down well in an industry where everyone wants to be a senior software engineer, a tech lead or higher...

I only wish that Informatics Technicial would be more prepared. And that Software Engineers/Licenciates would really be worth their salt.

Maybe that would be understanding the organs, the underlying physical processes those organs regulate, the chemicals that are released as parts of those processes, and the effects and composition of those chemicals, and if it is possible to induce similar effects through replacement chemicals in the way of medication.

Biology is not much different, though there schooling seems to do a better job at exposing the student to all the layers we know about.

On one hand I definitely agree that it's good to know lower level technologies and that's something I always ask people in the interviews. I think it's important because I know it.

On the other hand there is no end on how low you can go in the technology stack. Do you need to know how sockets works underneath? Low level network protocols? Do you need to know how hardware works because it probably influenced decisions that were made on the low level of the software stack? I know what socket is, but maybe the only reason I know it is because I'm old and I started coding at the time when raw sockets was the only way to implement network communication. When we have modern libraries and frameworks do we really need low level knowledge?

There have been a number of cases where I've had to rely on the ability to debug binaries directly at -O3, or to resort to Wireshark to get the dose of reality needed to challenge my [our] flawed mental model. If I hadn't been able to do those things, I'd probably still be there, pondering those defects, if not in body [due to declaring outright failure], at least in spirit.

More than tools they are products nowadays :-(

If you are running k8s, you should run a proper overlay network first.

I would question the assumption of having "modern" libraries, or at least rephrase it in terms of "performance" and "reliability":

Do we really have performant libraries (do they at least saturate the hardware)?

When it comes to networking, I think the answer is going to be more and more, no, we do not have performant libraries and software is being outpaced by hardware. Hardware is sitting idle thanks to our libraries.

Do we really have reliable libraries?

When it comes to storage, again I think the answer is no. For example, Linux RAID has long suffered from a plethora of faulty logic (pick a random disk sector as the canonical sector, split-brain etc.) and of course Linux RAID is not at the right layer to even have a chance of getting this right. And yet even ZFS lacks algorithms to detect or prevent split-brain. It's possible to detect (with 2 disks) and possible to prevent (with 3 or more disks), by taking quorum from the longest partially ordered set in the topology, but even ZFS (which is still awesome) is not reliable in this regard.

When it comes to networking, even TCP is not that reliable but has huge issues with bufferbloat. For example, a single person uploading a Gmail attachment on a 10mbps line will break the Internet for a second or two for everyone at your office (try it sometime - get everyone running "ping" and then upload an attachment). This is why Apple's background software update uses LEDBAT. Most libraries are not using LEDBAT or similar but are trusting TCP's congestion algorithms to do the right thing.

Are you a storage expert? Do not ignore how storage hardware is designed (even on a historical perspective) and frequently think about the system calls it is your responsibility to implement.

Are you developing a sidecar proxy? Be fluent in low level network programming, and be aware of the interface you are offering to the application.

Are you a manager? Focus on your own manager and objectives, but do not forget to empathically look at your team members.

And a frontend sitebuilder doesen't have to know anything about transistors, FinFET, system calls, and only the most curious would know about reflow and painting calls in a browser's rendering engine. Sure, the quest for performance always pushes people toward the metal, but from a website's perspective that's just the browser, and maybe TCP window tuning. (And sometimes someone finds that DNS was slow, but sometimes it was a bad BGP constellation, and there was nothing to do really. Or maybe the load balancer - or the backend - was overloaded because there was mutex contention in the kernel - or in the DB behind the backend, but that's again rather far from the usual sphere of investigation for "Frontend Engineers".)

The imporant point would be to be able to go deeper, if there's a need. Or go higher, more abstract. That's engineering (or software design), recognizing, analyzing, and solving problems.

I can't really blame the people who get by just knowing parts of AWS. They make great money by being a Cloud Consultant. It simply shows the low hanging fruits on the market. And how bad the demand side is when it comes to picking the experts.

You meant, there is a very strong demand ?

And yes, there's a demand for these specific skills, because big inefficient companies are willing to pay for just this, because their in house people are usually overworked or simply not allowed to do it themselves without an expert on site (or on call, or on skype).

And of course AWS is a big jungle of buzzwords itself, without clear documentation (and that's of course very important for them, they don't want to give away trade secrets, nor it should matter for the guarantees they offer, but their own custom nomenclature and sort of arbitrary abstractions over the standard Xen/KVM/qemu, VLAN, VXLAN, etc stuff makes it very hard to know exactly what's going on), so no surprise that there's demand for this. But as others said, sometimes these consultants have rather narrow skills.

And to add a bit more to this, generalists are few, and always quickly disappear from the market, so the holes/niches are plugged by whoever happens to be around.

Finally, I don't exactly blame these corps, it's just what makes sense nowadays. Need for extremely short turn around, instant risk-elimination, ASAP ROI, all point to "experts" and "consultants". No point in getting someone who understands the full stack top-to-bottom, when the project is in the initial phase and you just need EC2 and a DB on RDS.

Of course I get more satisfaction in "feeling" a "deep" and "true" technical folk.

I think these are never contradictory, at least I think software jobs allow for the flexibility to keep working on high-yield things while learning something for depth on the side. (Or vice-versa, if you live off your side job, and do/learn something deeper/longterm.)

Especially with network, virtualization, security, distributed systems (DB, storage), etc. the ideas that are trendy now are spin-offs of older things. (Docker is basically LXC with a sane UI/UX, which is just chroot done better, a'la BSD jails, which is just Solaris zones, which is just IBM mainframe partition shit. Similarly everything on the network, like overlay networking, VXLAN, Geneve, OpenFlow, etc. are just natural layering abstractions and applied trade offs that were considered silly a decade ago, or were simply not needed a decade ago. Now that north-south traffic is dwarfed by east-west traffic in every system [this means that north is the client, south is your "DB", and east-west just means services in your system, and since people nowadays use a lot more complicated systems, the inter-component = intra-system communication is huge, compared to the few MBs sent to the client].)

>Now that north-south traffic is dwarfed by east-west >traffic in every system [this means that north is the >client, south is your "DB", and east-west just means >services in your system, and since people nowadays use a >lot more complicated systems, the inter-component = >intra-system communication is huge, compared to the few >MBs sent to the client].)

Never thought about it, in those terms.

Sockets are generally accepted as the standard. If you want to stream data to another computer, that's pretty much the only way to do it. The solution has been decided already.

A lot of these frameworks live in a space that's still evolving. We, as a community of developers, haven't agreed on the 1 best way to do it. As such, there are multiple offerings. In these cases, I feel it is definitely necessary to now what's underneath, in order to better evaluate ones options.

Not completely true. I think it's possible to have a complete simplified image of a computer in ones head while working on the newest version of your web app, going into depth in this area where problems arise.

And usually it's the other way around than you might think. Sockets you learn once and can probably even apply in 40 years. The current hip web framework you can't even use anymore in the same company in 2-5 years.

What we discovered is that this meant they understood kubectl pretty well, and could write the yaml files. He didn't understand anything under the hood, so we had to re-discover everything.

And, ultimately, if you want to get a feel for how complex k8s really is, try and use their documentation beyond the very basics. It's contradictory, incomplete, and behind by a version of two sometimes. It was fun building a set of automated interactions with kubectl and its cousins from the web documentation, only to find it doesn't work because the actual binaries expect different arguments.

Or the API libraries if you're feeling particularly brave: a set of these horribly complex, dynamically generated files. Oh, some of them will be alpha, others beta, others beta2, etc. The version depends on the function you're using. Have fun!

Would you complain that you can't be a doctor without significant training or a lawyer without understanding the law? At some point, knowledge is required. Distributed applications require certain primitives and those can either be provided by Kubernetes or built from scratch, but they need to be understood to be used. There's no shortcut to that.

People tend to say Kubernetes is complicated because they don't understand the problems it's solving in the first place. If they did, it would be rather straightforward or they would recognize they don't need it in the first place.

That may be deliberate resume-padding. Same as most Hadoop implementations.

unless you have a single server with no availability your system is probably not "simple".

* what do you use for master election

* what do you use to deploy your stuff

* what do you use on the tcp level for HA (keepalived?)

* what do you use to keep your system up to date

these would be my first questions about your system. and if you would use k8s all of them could be answered with "kubernetes"

Not trying to be snide, but so far I haven't found k8s to be much more than primitives with which to build those things.

I've also only begun working with it recently, so I do not have much experience, and any guidance would be much appreciated.

what do you use to scale your databases:

databases can be scaled with statefulsets there is a good library that helps with that: github.com/zalando/spilo it is actually way easier than scaling over vms (especially on bare metal) since 1.10 you can even use block storage with local storage, so no need for a storage system. so basically k8s is just a good layer to scale such things, before we used patroni without k8s and it is way harder to actually do service discovery (more on that later)

what do you use for master election:

master election can be done with configmap or endpoints (actually github.com/zalando/spilo uses github.com/zalando/patroni for running postgresql and does master election over endpoints or configmap annotations) actually etcd also exists when you use k8s (actually it's the hardest thing to install when you deploy k8s)

what do you use to deploy your stuff:

well deployment/apps are extremly easy to use and do zero downtime deployments when you just change your image version

what do you use on the tcp level for HA (keepalived?): some folks at google created metallb: https://metallb.universe.tf/ it's actually a load balancer on top of k8s that is so simple to use (just run https://metallb.universe.tf/tutorial/layer2/ and it will work)

what do you use to keep your system up to date: well that's more tricky however ContainerLinux (formerly CoreOS) (not related to k8s) works very well for keeping the os up to date. you can even use https://github.com/coreos/container-linux-update-operator to only update one node at the time in a k8s env. however it also works without k8s, but you still need to run mostly container software on ContainerLinux.

If you are new to k8s and want to know more about "easy" to use on bare-metal stuff you should look into ContainerLinux ignition and kubeadm, it's really simple to get up and running from scratch, it's mostly just creating a nice cloud-config and run kubeadm once on the nodes you wanna install.

And it's good for deploying big systems with lots of moving parts (such a OpenStack, or any proprietary system, let's say a big e-commerce site).

And even if the problem is straightforward (just run these docker containers, just do this rolling upgrade, just do setup these nginx load balancers, just add these to monitoring), all of these are a chore, especially if done right. And config managers always fail at the premise of apply changes to a "known state" (or they spend an eternity checking the state), so why not fuse actual monitoring with config management, and a lot of nice concepts fall out automatically, like Deployments. No longer do you have to disable the alerting part of monitoring, or mark certain parts of the telemetry time series as irrelevant for SLA accounting because we were deploying (or cobble together scripts to handle this for you), and then evolve the whole system as the underlying components (the things that are being deployed) change.

Sure, k8s is not there yet, as the problem is big, complex, and hard to get right even in parts, but a lot better already than the simple one-man-army homegrown solutions. (IMHO)

Please don't miss the Cloud Native Trail Map, which is the front side: https://twitter.com/dankohn1/status/999727252135972864

They do write code but the technical job, per-se, is not that skilled.

At every level of abstraction they is a boundary that people may not choose (or have time) to pass.

I'm not saying everyone should be forced to learn about every layer, but these knowledge boundaries exist everywhere, and I'm not sure what can be done to improve the situation...

Kube is very clearly in the expansion phase of its lifecycle but I doubt it will stay this way over the long run.

Additionally, I think the fact that the overall categories are pretty well chosen is a sign of how decently understood the macro problems are and that there's a good reason to hope that you'll end in a place with 1-2 dominant players in each.

Then it became a commodity and now you can just use postgresql for a basic web app without knowing much about dbs.

Start adding scale and availability requirements in the mix, and you're back in need of specialized knowledge on indices, query planning, disk access patterns and db admin in general.

Same for cloud providers imho: you can throw an app online quicker than ever before, but as things get more complex you will still need an understanding of infrastructure, network, availability, security etc.

I keep meeting people who list say Postgres as a skill and when probed they admit they just clicked a button in Amazon RDS and that’s all they know about it. At some point it crosses the line into outright deception. This is the other side of the coin on why interviewing is so broken too.

I once applied for a job writing tools for embedded programmers that said they want experience in Angular, React, SPAs whatever. I'm working in embedded systems, so obviously 99.9% of people haven't even touched web apps, never mind the modern technologies. I actually got to the interview (despite no deception), because I knew someone working there. They said they actually don't have a single web app and are doing everything in C# (on desktop), and were thinking about making some web apps.

They still don't have any web apps.