They're complementary - LoadBalancer Services are L3, Ingresses are L4+. More often than not in-cluster Ingress providers (eg. nginx-ingress-controller) will in fact use LoadBalancers to actually route L3 traffic into the cluster in the first place.

Without being able to create LoadBalancer services there's no easy way to get any traffic into your cluster other than using NodePorts, and these have tons of shortcomings.

NodePort is a bad practice for a such case, you are right. You get additional unnecessary routing on CNI level between the cluster nodes just to get traffic into the Ingress controller pod, for example. The solution here is to use hostNetwork mode with the pool of workers solely dedicated to scheduling Ingress controllers. Traffic directly hits NGINX/Envoy/HAProxy/Traefik/whatever and gets into the cluster without additional intermediates. You need a load balancing solution for this pool of Ingress controllers, that's right, but as I said before, this setup gives you the flexibility to cook load balancing as you desire.

BTW, community NGINX Ingress controller is able to ingress L3 (TCP) traffic into the cluster.

> You need a load balancing solution for this pool of Ingress controllers, that's right, but as I said before, this setup gives you the flexibility to cook load balancing as you desire.

Sure, but this means that you cannot use LoadBalancers, which is painful. It means every payload has to be configured both at k8s level and then externally. That somewhat defeats the use of k8s as a self-service platform internally in an organization (other dev/ops teams need to go through a centralized ops channel to get traffic ingressing into the cluster if for some reason they can't use an Ingress).

> BTW, community NGINX Ingress controller is able to ingress L3 (TCP) traffic into the cluster.

Yes, but it's configurable via a single ConfigMap (which limits self-service if you're running a multi-tenant orga-wide cluster, unless you bring your own automation), and still you only have one namespace of ports, ie. a single external address for all ports - unless you complicate your 'external' LB system even further.

With all these caveats, I really don't understand why not just run metallb.

You lead me down a rabbit hole, thank you!

Do you know whether ingress controllers like nginx-ingress-controller honor the readiness status of pods from a service and so only sends traffic to ready pods from the concerned deployment?

They send traffic to the Service that exposes your Pods, so yes, it only gets routed to Endpoints that work according to readiness tests.

I remember seeing documentation on this a while ago, but I can't seem to find any documentation from nginx-ingress that talks about how nginx-ingress uses readinessprobe or livenessprobe.

I was looking into this because if I have a pod with 3 containers, if 1 container wasn't running, nginx-ingress stopped serving traffic. I actually want to continue serving traffic if a certain container is still running, not all 3 running necessarily

Do you know or have any documentation on how nginx-ingress actually does its readiness/liveness checks?

ingress-nginx doesn't do readiness/liveness checks itself; the kubelet does[1].

If any of the containers in a Pod aren't ready, the endpoints controller[2] removes the Endpoint object corresponding to that Pod. ingress-nginx watches these Endpoints objects[3] to determine which Pods it should send traffic to.

Edit: As to your use case, I think you should remove the readiness probe from that one container you don't care about.

[1]: https://kubernetes.io/docs/tasks/configure-pod-container/con...

[2]: https://kubernetes.io/docs/concepts/overview/components/#kub...

[3]: https://kubernetes.github.io/ingress-nginx/user-guide/miscel...

Third paragraph in the documentation says that a Pod as a whole is only considered ready if all containers are ready:

https://kubernetes.io/docs/tasks/configure-pod-container/con...

hmm makes sense, thank you!

Not all services exposed to the outside are HTTP(S).

Yes, Ingress resources implicitly assume that you want to get HTTP(S) traffic into the cluster, but as for example, ingress-nginx is able to expose gRPC via additional annotations and generic L3 via related ConfigMap.