thedevopsguy

I find it interesting that many people seem to conflate the complexity of managing infrastructure and services with K8s.

K8s is complex because managing distributed services is. Not using it doesn't mean it goes away. The complexity migrates and ends up being bundled up in a separate tool or a runbook process or some script.

It's hard to maintain because the tools and apis are different from what some engineering teams are accustomed to using. Building an in-house tool gives them a warm fuzzy feeling and comfort that they can handle problems when they appear due to familiarity with their own code and design choices.

It's a fair trade off. I do wonder how much of the time spent doing this exercise could have been spent on K8S training.

I do feel that the K8S community do downplay how much a PITA k8s configuration can be and that the perceived robustness of cloud-managed K8S isn't up to scratch for something this complex.

Your standards aren't too high but I think you must realise that this is a cultural problem with little hope of changing. Even if the push comes from the CTO, it will take years for change to happen, it will require new hires and bringing new blood into the engineering leadership.

If you do want to take the challenge (which I strongly discourage you from) you'd need to collect data to build your case, quantify the time and human cost from issue/jira to code landing in prod to number of incidents/bugs. The instrumentation to do this will be a fairly chunky piece of devops work. Frame the data in light of your competitor's ability to iterate their products and so on. When it's collected and presented it can be quite compelling and people will listen.

It's only at this point you'll be able to present the problem to management in way that they understand. You know and I know that this is a cultural problem first, then a process problem and lastly a technology problem. The amount of work to effect this kind of organisational change, even in a small engineering company is immense. I don't know your motivations are for staying, if it's the domain or the money but if this something that bothers you then this is the best piece of advice I can give you:

Run, head for the hills, and don't look back.

I'm referring to having the same testing, deployment,packaging,versioning policies etc being consistently applied across projects within the same repository not deploying, testing and releasing together.

It's the drift and inconsistencies across these concerns across projects that makes deployment and operations less predictable.

Without knowing more about their architecture it is difficult to comment beyond the conclusion Alexandra Noonan came to, stated at the beginning of the article. It looks like to me that the architectural assumptions were changing too quickly due to the demands of a fast growing business. Having all their code in a single repository means that they can control dependencies, versioning and deployment centrally, it gives them central control of their software development lifecycle. I can't see how they could not have had the same benefits of the monolith if their microservices existed in a single repo to begin with and had the appropriate tooling to enforce testing, versioning, deployment across all services in the repo. I guess this is the whole monorepo debate and tooling.

This article for me is more about the complexity of managing a large team across different sites where the architecture needs to change rapidly when modularity is absent. They did get a measurable benefit around performance, though. I wonder if Alexandra will comment on the challenges of running a team in an environment of this complexity?

I've been an early adopter of docker. Used Compose when it was still called Fig, used and deployed kubernetes beta up to version 1 for in-house PAAS/heroku like environment.

Must say I do miss those days when K8s was an idea that could fit in your head. The primitives were just enough back then. It was powerful developer tool for teams and we used it aggressively to accelerate our development process.

K8s has now moved beyond this and seems to me to be focussing strongly on its operational patterns. You can see these operational patterns being used together to create a fairly advanced on-prem cloud infrastructure. At times, to me, it looks like over-engineering.

Looking at the borg papers, I don't remember seeing operational primitives this advanced. The develop interface was fairly simple i.e this is my app, give me these resources, go!

I know you don't have to use this new construct but it sure does make the landscape a lot more complicated.

Kubernetes by itself may be daunting for most teams.

But I'm not sure I understand the backlash. Once you've built your application and it's been packaged (containerized) and deployed why would anyone care how its run. Also running a container in production and orchestration seem to be conflated somewhat in this thread and the use cases are very different.

You can think of Kubernetes as an Automated SysAdmin . This is a bit reductive I know but it is useful to think of this way. You ask the sysadmin to run something for you and they tell you how to package it (tgz, war, zip etc) and they run it for you on hardware.

The level of engagement that a dev has with getting his app running on hardware is no different to that of dealing with a sysadmin and with the admin requesting that your app is packagedin a container.

Kubernetes out of the box will give you most of this functionality as long as you keep state outside of the cluster. There are also options on how to make the experience smoother. There also these tools to help too:

* Openshift * Kubernetes + Rancher * Mesos

If you need orchestration and scheduling. I am a little perplexed.

My comment wasn't too clear. I'm not talking about just running docker cli on Windows or using windows 10 containers. Am yet to see a coherent story on how to write say a Java application containerize it, deploy it on Windows and Linux without having dealing with differences in each OS container solution.

If there is, I'd like to learn about it.

Hadn't thought of it from this angle. Docker's only chance of survival is to have a cross platform container. Something that works on Windows and Linux.

Perhaps Docker's only play is to fold into Microsoft to achieve a cross platform solution. Microsoft does have Brendan Burns now.

It's all about Openshift. Redhat developers have actively contributed to Kubernetes for about two years now.

Now they'll own the entire stack and have a great integration story for enterprises. Even though containers have been around 3+ year's in the form of docker, corporations still don't have a scooby on how to integrate their existing deployment and development workflows.

The docker ecosystem is hard to follow. Like you've just mentioned there are multiple solution to each problem. Docker based solutions for orchestration(Swarm), storage (v1.9) and networking (v1.9) overlap with the offerings from Kubernetes,Mesos, Flocker and whole bunch of others.

It's hard to know whether to wait for Docker to provide a solution or to use something that already has momentum. Take networking for example. Solutions have been bandied about for the last year or so and only now do we have something that's production ready. Do I rip out what I already have for something that is docker native or do I continue with the community based solution.

Storage (Data Locality) also follows a similar path. Kubernetes provides a way for making network based storage devices available to your containers. But now, with the announcement of Docker v1.9 do I go with their native solution or something that has been around for ~6months longer?

I've been working with these technologies for the past year and it has not been easy building something that is stable with a reasonable amount of future-proofness baked in.

Hi Ctex,

It's neat that beluga is written predominantly in bash but it's also difficult to see what Beluga actually does. Right now it looks like it sets up an environment for a docker compose app to run. Similar to docker machine but also solvable by tools like ansible, salt or python fabric. Kubernetes and Mesos are solving different problems. They manage and orchestrate services and add ons may also help with repository management.

My view may come from a lack of understanding of the tool's primary use case. Deeply interested in new developments in the docker ecosystem. Could you please update your current documentation with a few examples.

Wasn't aware of their move. A microservices architecture then makes the technical conversation about the service, it's latency, performance rather than implementation details.

Is it such a bad thing to have many different languages in play as long as the SLAs are met?

Or move everything to a microservices architecture and use the best suited language for each subsystem. It would force both camps to engineer for composability and open service apis. My tupence.

Log analytics is a big topic so I'll hit the main points. The approach you take to logging depends on the analysis you want to do after the log event has been recorded. The value of the logs diminishes rapidly as the age of the events get older. Most places want to keep the logs hot for a period ranging from a day to week. After that,the logs are compressed using gzip or Google snappy compression. Even though they are in a compressed form they should still be searchable.

The most commont logging formats I've come across in production environments are:

1.log4j(java) or nlog(.NET)

2.json

3.syslog

Tools that I've used to search ,visualize and analyse log data have been:

1.Elasticsearch, Logstash and Kibana (ELK) stack

2.splunk (commercial)

3.Logscape (commercial)

Changes to the fields representing your data with the database approach is expensive because you are locked in by the schema. The database schema will never fully represent your full understanding of the data. With the tools I've mentioned above you have the option to extract ad-hoc fields at runtime.

Hope this helps.

Missed the extra 9. Read and re-read. sincerely thought he called two identical categories rare and very rare.

Long day

thanks.

My brain is shutting down and I can't parse this phrase.

[1] "If you lack an antigen that 99 per cent of people in the world are positive for, then your blood is considered rare."

[2] "If you lack one that 99.99 per cent of people are positive for, then you have very rare blood."

Surely the author is saying the same thing here?

There is some confusion around the article and it may be because of the way it is written. but here's a brief summary. Hope this helps to clarify:

* The theory/hypothesis is not saying avoid pronouns.

* It's about relative frequencies not absolute.

* The pronoun frequency is looked at in different scenarios:

   1. between two people who don't know each other 

   2. between two people who do know each other 

   3. pronoun frequencies of an individual in a diary, blog over a period of time.

* The frequency of pronouns in spoken or written language is an unconscious activity. It's something that is hard to fake, unlike body language.

* The words being compared/counted are primarily social identifiers vs determiners and articles.

According to Pennebaker status is negotiated at the beginning of a new interaction when the perceived status of the participants is unclear. The frequency of the filler words mark the status/role that has been adopted by the participants. Which is clearly marked in the email examples given.

The article does not imply that this is a learned skill, since the whole theory is based on function words which are accessible at even the most basic proficiency of a language. It is merely describing what happens to our language when we enter a social interaction.

Note the words used aren't necessarily important but rather the function of the words are, i.e are they referencing social objects ( the role of pronouns) or referencing concrete or abstract non-social objects.

Haven't seen flats sell as quickly as they are now. Living a 3 minute walk from Liverpool Street station and the flat I wa renting sold by the second viewing. It was on the market for three days but things are cooling off.

When I first starting working with docker I'd use the

  docker inspect $CONTAINER |  grep -i VAR  

pattern alot until I discovered that you can do use the container name and do go with:

   docker inspect --format '{{ .NetworkSettings.IPAddress }}' replset1

Service discovery and docker is still a pain point with the technology. Serf [1] and etcd [2] are tools that manages a cluster of services and helps solve the problem described in the article.

[1] Serf is by the guys behind Vagrant. - http://www.serfdom.io/

[2] etcd - http://coreos.com/using-coreos/etcd/

Ecstasy and water poisoning are related. Ecstasy makes users extremely thirsty. Water poisoning related deaths at raves have been reported. About 6 L is enough to put a lot of people in a very dangerous place.

Using a log shipper is probably the best workaround in this scenario

1.) logrotate at the end of the day 2.) have your logshipper watch docker log folder for each container 3) Log shipper ( or collector) ships files as they are updated to a central server and you are free to archive or delete as you wish.

Many of the central logging systems can detect rolled logs, so this set up is not much of a stretch.

I do agree more configurable logging is a bit of an oversight, especially for something like Docker.

Remember this book too. It's a classic text in AI. I think what they were trying to say in the intro is that the approach has a significant effect on how much effort is wasted. The engineering effort required to get a plane of the ground using moving wings is considerably greater than using rotational parts like propellers. Imagine how many decades would have been wasted to scale an orthinopter to handle the load of a passanger jet.

This makes me think that AI research isn't going to be a gradual process of research being built on other research but it will be a eureka or an ah-ha moment that changes everything.

Elastic Search make their money from Enterprise support licenses. There are so many ways of installing it , scaling and optimizing depending on application. Don't forget the training around. You have IBs like Goldman Sachs using it for log analytics. So the services around that are needed and real. I don't think they will need an enterprise version.

Didn't games like Duke Nukem 3d, post-Doom, Wolf games use what was called a scanline algorithm? Drawing the lines in the players FOV as you just stated and then using a clipping algorithm to keep unwanted information out. I think raycasting would be a type of scanline algorithm, but the technique is less primitive since your not using a proper 3d engine. It's a long time since I programmed games like this so am now a bit fuzzy on the specifics.

As humans we primarily make time for what we really, really want to do over what we should. Before getting angry at someone for being late take a step back and assess the situation objectively. The person may have been late because they probably weren't that interested in seeing you, they may have had something more interesting to do or had an interesting conversation going on.

Walk away after 15 ( or 10) minutes because you should have something better to do with your time than wait around hours for someone, right? There needs to be a negative consequence of the bad behaviour and it should be communicated through your actions rather than your words. Words will always fall on deaf ears.

I remember a friend who had a problem with a very tardy friend from one of the warmer EU countries and started to leave habitually after 10 minutes of tardiness. After a couple of times he got the idea and even started arriving early to the meetings.

You don't lose anything by doing this. It's a good strategy since it reveals the nature of your relationship with that person. If you don't like the revelation, fix it in a socially intelligent manner or exit the relationship. ...