Using Single Load Balancer across multiple namespaces in Kubernetes

Photo by Jeremy Bishop on Unsplash

One Loadbalancer to rule them all? you heard it true, It's achievable!

AWS LoadBalancer Controller

Until a couple of weeks ago, we were creating a load balancer for each namespace ( by default from AWS), which was a waste of resources and money. Hence we thought how can we use a single load balancer across all the namespaces.

Here’s an example of before migration, how ingress looked like for default namespace:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-default
  namespace: default
  annotations:
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80,"HTTPS": 443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: instance
spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /*
            pathType: ImplementationSpecific
            backend:
              service:
                name: deployment-A
                port:
                  number: 80

Now, let's take a look at ingress at ArgoCD namespace :

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-argocd
  namespace: argocd
  annotations:
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80,"HTTPS": 443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: instance
spec:
  rules:
    - host: argocd.example.com
      http:
        paths:
          - path: /*
            pathType: ImplementationSpecific
            backend:
              service:
                name: argo-server
                port:
                  number: 80

• Now, the problem with the above configuration is, by default AWS load balancer controller will create ALB for each namespace.
• The cost of creating ALB for each namespace is ~ $17.99 and data transfer charges are $0.02 per GB. If you got a lot of namespaces, the cost will be a lot over the year. This is not how Loadbalancer supposed to be used as, each ALB can handle a lot of loads easily.

After hunting for a while, we found that the solution lies in using an ALB annotation called group.name on the Ingress object.

The flow

• let’s say you got 5 different ingresses with 5 ALB in the backend in different namespaces, choose one ingress and add the annotation :

alb.ingress.kubernetes.io/group.name: <ANYTHING_MEANINGFULL>

• Now, apply the same group name annotation to all the ingress objects in all namespaces.
• AWS LoadBalancer controller will use the first ingress Loadbalancer for all ingress objects and all other 4 ALB, in this case, will be removed.

Below is an example for 2 different namespaces with a single ALB :

Default namespace ingress :

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-default
  namespace: default
  annotations:
    alb.ingress.kubernetes.io/group.name: staging-lb
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80,"HTTPS": 443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: instance
spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /*
            pathType: ImplementationSpecific
            backend:
              service:
                name: deployment-A
                port:
                  number: 80
--- 

ArgoCD namespace ingress :

#argocd namespace ingerss
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-argocd
  namespace: argocd
  annotations:
    alb.ingress.kubernetes.io/group.name: staging-lb
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80,"HTTPS": 443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: instance
spec:
  rules:
    - host: argocd.example.com
      http:
        paths:
          - path: /*
            pathType: ImplementationSpecific
            backend:
              service:
                name: argo-server
                port:
                  number: 80

• Notice how both ingress objects are using the same annotation with the same Group name.

Whenever next time you need to create a new ingress in a different namespace, you can just add the same group.name annotation and it will work flawlessly.

NGINX Ingress Controller

If you are in any other cloud, let's say DigitalOcean, its cloud provider controller doesn’t have any custom LB controller. But don't get sad there buddy, NGINX to the rescue.

• You can leverage the NGINX ingress controller to create & manage LB for you.
• For installing the NGINX ingress controller, you can refer to their documentation which has all the steps to install it.
• Let's say you got 2 different namespaces with 2 different ingress objects same as above one for default and one for ArgoCD.
• The solution is to add ingressClassName: nginx in the ingress object Spec section in each ingress file.

So the final ingress file looks like this for the default namespace:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-default
  namespace: default
spec:
  ingressClassName: nginx
  rules:
    - host: chartmuseum.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: chartmuseum
                port:
                  number: 80

Argocd namespace Ingress ingress looks like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-argocd
  namespace: argocd
spec:
  ingressClassName: nginx
  rules:
    - host: argocd.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argocd-server
                port:
                  number: 80

Happy Loadbalancing :)

Originally published at https://tanmay-bhat.gitlab.io.