> Kubernetes on the other hand needs atleast 12 yml files

You don't need to split your manifests into multiple files. You can use a list:

    apiVersion: v1
    kind: List
    items:
    - apiVersion: extensions/v1beta1
      kind: Deployment
      ...
    - apiVersion: v1
      kind: Service
      ...
    - apiVersion: v1
      kind: ConfigMap
      ...

Edit: Or YAML's "multiple document" syntax:

    apiVersion: extensions/v1beta1
    kind: Deployment
    ...
    ---
    apiVersion: v1
    kind: Service
    ...
    ---
    apiVersion: v1
    kind: ConfigMap
    ...

Then you just use "kubectl apply -f myapp.yml" to create or update.

Re ingress, I agree that it's probably the weakest point of Kubernetes at this point. It's particularly weak when it comes to internal load balancing. When you have lots of services that should only need to be available internally, you'll want to have an internal ingress; I don't know about AWS, but Google Cloud Platform's internal load balancer doesn't support pointing at Kubernetes [1]. I haven't found a better option than to run Traefik as a DaemonSet and rely on round-robin DNS (aka poor man's HA).

[1] https://github.com/kubernetes/ingress/issues/112

Or just put --- between various stanzas in one file: https://kubernetes.io/docs/user-guide/managing-deployments/#...

(Google now has a L3 internal load balancer, but not a L7 LB. It's fair to say that this will be made much easier this year.)

I really think the community ought to adopt this as best practice and actively discourage (or even deprecate) the List type. It's just too much noise and zero benefit IMO over raw slices or serialization as arrays (or document lists for YAML).

As the author (blame me) of the List type, the primary advantage is it needs no special logic in JSON for processing. New line separated JSON is wierd for a lot of libraries, and in the future we want to have endpoints that allow bulk creation / bulk apply.

That's good news. ILB ingress integration is forthcoming, then, I hope?

True! I keep forgetting the multi-document support in YAML.

>I haven't found a better option than to run Traefik as a DaemonSet and rely on round-robin DNS (aka poor man's HA).

you could try linkerd. It is built with precisely this usecase in mind

Also, it was not just a question of number of files. The size of my docker-compose.yml is about 30-40 lines or so.

I think Helm [1] looks promising in this regard.

Helm "charts" are templates that generate Kubernetes manifests. To install a chart, you provide values (for which there are defaults). For example, here [2] is the chart for PostgreSQL. The default values are in values.yaml, and the templates for each manifest are in the templates folder.

Conceptually, this even cleaner than Docker Compose, because there's zero data that isn't specific to your install: Everything else defaults to a pre-defined default.

So in the same way that the official PostgreSQL Docker image is general-purpose, you can define a completely general-purpose chart that can be used by anyone. You customize it by providing overrides.

The only downside is that you still end up with Kubernetes manifests. It's not an abstraction, it's an automation tool.

[1] https://helm.sh

[2] https://github.com/kubernetes/charts/tree/master/stable/post...

You are absolutely right that getting traffic inside the cluster is probably one of most difficult tasks in Kubernetes. The standard ingress objects are too simple when you have a complex deployment. So, we wrote our own Ingress Third Party Resource to fit the needs of our deployment. We use HAProxy. We have been using this in prod for almost a year. You can find it here: https://github.com/appscode/voyager

If you find any issues or have questions, please file Github issues.

I found this last week and I'm very impressed because it solves a lot of the problems we had with existing ingresses.

I really like how you guys decided to do your own 3rd party manifests instead of using annotations.

I'm going to play more with this :-)

this is awesome!

please please submit this to kubernetes-incubator

Thanks for the kind words. The incubation process seemed too complicated (process heavy?) for small teams like us. But I will look into that again.

Author here

> The first thing I do when I look at kubernetes tutorials is look at ingress or load-balancing... and usually I find it absent.

Yup! I dedicate a few minutes to this in the grab lesson because how important it is.

> Docker Compose yml form 3.1 is brilliant - other than my docker files, i need a 30 line yml file to deploy my whole stack across 3 machines. Kubernetes on the other hand needs atleast 12 yml files ("deployments" and "services"). I'm not counting another 4 yml files for statefulsets and persistentvolumeclaim since there is nothing equivalent in docker swarm

Interesting point. You may like helm (https://helm.sh) as a tool for deploying Kubernetes applications.

actually, docker compose file is so awesome that there is now a top level incubator project called kompose that is trying to provide drop-in compatibility. https://github.com/kubernetes-incubator/kompose

you should check out the #kompose channel on slack.

> Docker Compose yml form 3.1 is brilliant

What's nice about ECS is that it lets you deploy using Docker Compose files (though you often make minor modifications for ECS)

    ecs-cli compose --file ecs-compose.yml create

However, there is the kompose project to bring Docker Compose to Kubernetes (it's still in Kubernetes Incubator at this point):

http://blog.kubernetes.io/2016/11/kompose-tool-go-from-docke...

https://github.com/kubernetes-incubator/kompose

Maybe you or someone else can help me understand the use case for a load balancer. I have an application where it has a very high rate of ingress traffic. So much so that while a load balancer sounds like it's ideal, a single node (the balancer) would not be able to handle the traffic. Is the load balancer only useful for applications with a low ingress rate that you're trying spread across many servers?

How much traffic are we talking about? Google Cloud Platform's load balancers (which sit outside of Kubernetes and are replicated geographically) are designed for quite a high load of traffic.

Either way, your client has to know what pod (group of Docker containers running your app) to send traffic to. The pods run on hosts and ports that aren't fixed in time. So if you want some kind of client-resolving load balancing without involving a central bottleneck on the server, you'd have to transport that inventory of pod-host-ports to the client, and then either provide a host-based proxy or open the ports on the nodes themselves. And of course you'd have to build the balancing/failover logic into the client (which would either have to be randomly distributed, or rely on resource utilization info fetched from Kubernetes).

That may be great for some narrow use cases, but for most applications, the load balancing pattern is much simpler.

Do you know the details of that? On their site they talk about balancing in terms of queries per second on big query. What is that equivalent to in terms of Gbps?

Without a load balancer, how do you currently spread the load over multiple nodes in your app?

If you have control over the sender, you can, right? That's my use case and maybe that's why I don't see the value.

I see. A lot of us I think assume we are talking about web apps or mobile apps where you don't.

Load balancers are particularly useful when it comes to cloud applications because they let you keep your routing logic server-side, meaning that you can make adjustments quickly without needing to release a new client version.

Your load balancer definitely shouldn't be a bottleneck - you can deploy multiple LBs to handle the traffic and provide redundancy, though you need to use other methods to balance between them (DNSRR, anycast, etc).

great question!

kubernetes pretty much mandates either an LB because it is offloading the ingress job to a machine which can talk to the outside world and the VPC simultaneously. Otherwise you either get into complex hostport/nodeport/hostnetwork semantics... or use an ingress.

In the k8s world, there is not many simpler alternatives.

Could you provide some more detail about your statement that "nginx-ingress will lose source-ip info"?

I'm currently working towards a k8s-on-metal system using nginx for ingress and was led to believe that I could use either X-Forwarded-For or proxy_protocol to preserve the originating IP address. Is the traffic bounced through an SNAT rule on it's way to the ingress controller or similar?

you should google for proxy-protocol. so here's the thing - nginx cannot INJECT proxy protocol. I'm not an expert in nginx.. but atleast the versions used in k8s couldnt.

It can read/pass-through proxy protocol headers. So if you have haproxy in front of nginx.. it will work fine (or ELB/GLB).

Which is why there's another beta ingress - the "keepalived-vip" repo in k8s ingress. However, not a lot of people seem to be using haproxy primarily because haproxy cannot have a zero-downtime deploy.

A couple of weeks back on k8s slack, I was discussing using Github multibinder to solve that one problem and move ahead with haproxy ingress.

protip: please join the #sig-onpremise channel on slack. There's not a lot of momentum for bare metal deploys otherwise.

https://githubengineering.com/glb-part-2-haproxy-zero-downti...

I'm familiar with proxy-protocol, and while I haven't injected it with nginx, it seems that it does support it: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.htm...

For reference, the feature was added in nginx 1.9.2, and the ingress-nginx controller appears to use nginx 1.11.9: https://github.com/kubernetes/ingress/blob/0.9.1/images/ngin...

Regardless, would I be right in assuming that your issues are mostly related to the ingress of TCP streams? My usecase is almost exclusively HTTP, so I might be missing the worst of it.

Thanks for the pointer to the #sig-onpremise channel, I'll be sure to check it out. Been meaning to look into the SIGs since reading https://coreos.com/blog/self-hosted-kubernetes.html

please double check - nginx's support for proxy protocol is to be able to read it. it cannot support injecting it. if nginx is the first thing your traffic hits, it will not inject proxy protocol headers. Again, I will not stake my reputation on this... but I remember going deep into this about 2 months back.

if you are using nginx ingress, then you need to move all of your existing config into their own config file format. Since setting up full TLS internally in kubernetes is buggy, I didnt want to terminate my traffic on the ingress. Also I really did not want to move my (fairly complex) nginx configuration to the ingress.

Which is why I'm a big believer in haproxy ingress - it was designed to interface with nginx.

However, you are already one of the "enlightened" ones. You actually do know what an ingress is. A significant amount of k8s slack traffic is "how do I setup a loadbalancer/how to make the cluster available to the outside world"

nginx can inject proxy_protocol - I know because I've enabled it by mistake! The kubernetes nginx ingress controller doesn't support it though. I think that is because most people use HTTP headers (X-Forwarded-For) instead. But if you do want it, you should open an issue explaining the use case so we can check our assumptions!

@justinsb - ur the expert here and i will not dispute this :D

I still believe that there is NO better tool to use kubernetes than kops. I now use the 100.64 subnet that you guys figured out ("carrier grade NAT" seriously?) in docker swarm.

but genuine question - can u show me how ? Because I honestly went crawling into the nginx codebase and could only find client side handling of proxy protocol.

I found NO place where it showed how it could be injected and chain reverse proxies together (that is what it was invented for in the first place).

P.S. just a quick google reveals this link which sort of confirms my suspicion - http://blog.haproxy.com/haproxy/proxy-protocol/

It's the same proxy_protocol keyword for both, which is why this is so confusing. In the listen line it means "remove proxy protocol from the inbound connection", as a top level directive "proxy_protocol on" on the server it means "add proxy protocol to the outbound connection"

This commit should show the difference: https://github.com/kubernetes/ingress/commit/6fa461c2a7891b4...

This is the nginx function AFAICT: https://github.com/nginx/nginx/blob/b580770f3afaeec48a15cb8c...

Looking at that though, maybe it only works with SSL passthrough... but that is the typical use-case for using proxy protocol instead of X-Forwarded-For

What's wrong with using the X-Forwarded-For/X-Forwarded-Proto headers as usual? Or do you mean for non-HTTP protocol?

Yeah, looks like he's talking about L4 proxying. Check out http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt.

Couldnt agree more with the ingress sentiments