How can I get everything to use the same load balancer on DigitalOcean?

[Trenton]

I have a LoadBalancer service, and an Ingress that routes to various ClusterIP services. I've given both of them the following annotation:

annotations:
  kubernetes.digitalocean.com/load-balancer-id: 'myproject-dev-lb'

A new load balancer was created, although I see no reference to myproject-dev-lb on it. The LoadBalancer service (a Pulsar server) works fine, but my ingresses have stopped working. When I kubectl get my ingress, I get what looks like a previous IP address (might just be similar). When I describe it, it says it's not found. If I delete it and recreate it, it gets the same IP address. The address is pingable, but not listening on port 80. The created load balancer only has a rule for port 6650 (Pulsar).

I'm not really familiar with the relationship between Ingress, load balancers and services.

[kalyn]

A service is a way to route traffic to an internal IP. By internal I mean inside Kubernetes. I mean you don't really point it to an IP, you point it to a deployment or w/e and Kubernetes takes care of the IP and routing traffic for you to the pods that make up that deployment.

Now not all services work the same. ClusterIP will create an internal hostname and then route traffic to the Pods's IP. This make the service only reachable from other pods inside the cluster.

Then you have NodePort, which opens a port on all worker nodes and routes the traffic to the deployment/pods or w/e. This exposes the service externally. So now it's reachable by a worker nodes IP and whatever port was selected by the service. You can also hard code a port but I wouldn't recommend it.

But who wants to deal with IP's and port numbers? Thats lame. We want hostnames. you know like myapp.dev.coolcompany.com or w/e.

That is where a Loadbalancer comes in. If you have a cluster running on a cloud provider then your cluster is likely already setup to use a LoadBalancer service. This service does two things. First it creates a nodePort service and then it configured a load balancer on your cloud provider to point traffic to all worker nodes and the NodePort for your service.

When your LoadBalncer is created it likely came with a public DNS name to reach it by. Now you can just create a DNS record to point from myapp.dev.coolcompany.com -> my.lb.dns.cloudprovider.com. Now your app is reachable externally and without knowing about IP's and ports.

Everything I just explained is explained even better and more accurately by the Kubernetes https://kubernetes.io/docs/concepts/services-networking/service/ . You should probably give it a read.

So what's an Ingress? Well an Ingress is like creating a "LoadBalancer" but inside the cluster. We call this the Ingress Controler and there are a ton of options like Nginx, Apache and Traeffik to name a few. This likely does not come included by default on your cluster. You will need to create and configure an ingress controller before you can use the "ingress" API object.

An ingress controller works like any other deployment running on your cluster. You will need to expose this application externally. Most often using a "LoadBalancer" Service. Confused yet? It will look like this: ExternalLB --> IngressLBService(NodePort) --> IngressController

Once your Ingress Controller is configured you can then expose your deployments externally without using a "LoadBalancer" service. You can instead use a "ClusterIP" service + an "ingresss" object configured to point at the service. It will look like this: ExternalLB --> IngressLBService(NodePort) --> IngressController --> DeploymentService(ClusterIP) --> DeploymentPods

As you can see the "LoadBalancer" service and IngressController work very similarly. So why do you need an IngressController?

Well if your application stack is simple then you probably don't. Some of the cloud providers "LoadBalancers" are pretty feature rich. Like on AWS. But some of these Ingress Controllers offer some pretty nice features. Like path bases routing, integrating with Authentication providers, Automatic TLS certificate provisioning and tracing+montioring etc. Basically, if you need these features and your cloud provider's Loadbalancer doesn't have these things then an Ingress Controller is the best way to get them.

Another tiny advantage is that an Ingress can help abstract away the cloud provider. Meaning less vendor lockin. If I have an application that is exposed externally via an Ingress controller then I can re-deploy that application on any cluster on any cloud provider as long as that cluster has an ingress controller.

If I'm depending on a "LoadBalancer" service then its likely that service is configured with a bunch of Annotations and that is often not compatible with other clouds. Meaning I cant just redeploy that application to another cloud provider. This use case isn't very important to many people and it wouldn't be a big deal to fix a service to use a different cloud LB but it is something to consider.

You can find a lot more in the Kubernetes https://kubernetes.io/docs/concepts/services-networking/ingress/ .