Installing Traefik Enterprise Edition with Multiple Control Nodes on Kubernetes with YAML Files¶

This installation guide is for experts who want to fine-tune their TraefikEE (Traefik Enterprise Edition) installation.

It covers how to install TraefikEE using Kubernetes YAML files as a Kubernetes Ingress Controller.

Kubernetes Knowledge

Assistance with configuring or setting up a Kubernetes cluster is not included in this guide. If you need more information about Kubernetes, start with the following resources:

Requirements¶

The traefikeectl tool
A Kubernetes cluster:
- Supported versions: 1.10, 1.11, 1.12 and 1.13
- RBAC enabled (recommended),
- Access to the Kubernetes API with kubectl, with the ability to create and manage namespaces and their resources.
Bootstrap and control node pods can reach https://v3.license.containous.cloud.

Create the Namespace¶

Create a namespace named traefikee to host the TraefikEE installation:

Command

kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.2.1/kubernetes/multi-cn/namespace.yaml

Expected Output

namespace/traefikee created

Prepare Role Based Access Control (RBAC)¶

Create the RBAC objects to allow TraefikEE pods to interact with the Kubernetes API:

Command

kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.2.1/kubernetes/multi-cn/rbac.yaml

Expected Output

serviceaccount/traefikee created
clusterrole.rbac.authorization.k8s.io/traefikee-role created
clusterrolebinding.rbac.authorization.k8s.io/traefikee created
serviceaccount/traefikee-bootstrap created
clusterrole.rbac.authorization.k8s.io/traefikee-bootstrap-role created
clusterrolebinding.rbac.authorization.k8s.io/traefikee-bootstrap created

Note

The following Service Accounts are created:

serviceaccount/traefikee, authorized to access the Kubernetes API actions required for TraefikEE.
serviceaccount/traefikee-bootstrap, authorized to manage secrets for TraefikEE installation.

Create the Service¶

Create the services to allow network access to the TraefikEE cluster:

Command

kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.2.1/kubernetes/multi-cn/data-node-external-service.yaml
kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.2.1/kubernetes/multi-cn/control-node-internal-service.yaml
kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.2.1/kubernetes/multi-cn/cluster-internal-api-service.yaml

Expected Output

service "traefikee-lb" created
service "traefikee-control-nodes" created
service "traefikee-api" created

Note

The following services are created:

traefikee-lb handles the traffic of the ingress rules for your applications.
traefikee-control-nodes handles the traffic for control nodes internal communication.
traefikee-api handles the traffic for TraefikEE's API internal usage such as metric collection or Web UI.

Create the Bootstrap Node¶

Installing behind a proxy

In order to install a behind proxy, you must define the HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables for the bootstrap node. As the bootstrap node will make requests to the kubernetes API server, the NO_PROXY variable must contain its IP address. It can be found using kubectl get service kubernetes.

containers:
  - name: ...
    # [...]
    env:
      - name: HTTP_PROXY
        value: http://127.0.0.1:3129 # Your HTTP proxy URL
      - name: HTTPS_PROXY
        value: http://127.0.0.1:3129 # Your HTTPS proxy URL
      - name: NO_PROXY
        value: 100.64.0.1 # IP of the kubernetes service

The "Bootstrap node" is an ephemeral control node, only responsible for initializing the cluster with your license information.

Download the file bootstrap-job.yaml:

Command

curl -sSLO \
  https://s3.amazonaws.com/traefikee/examples/v1.2.1/kubernetes/multi-cn/hooks/bootstrap-job.yaml

Insert your license key in the file bootstrap-job.yaml:
- Search for line --licensekey=fakeLicense
- Replace the value fakeLicense with your license key

Bootstrap stopped after 5 minutes

If the control plane is not initialized within 5 minutes, then the bootstrap node will stop automatically. To avoid this, you have to tune both following parameters. For example, let's wait 600 seconds:

activeDeadlineSeconds: 600 (in the section spec)
--timeout=600 (in the section spec/templates/spec/containers/args)

Save the file

Create the Bootstrap node to initialize the cluster:

Command

kubectl apply -f ./bootstrap-job.yaml

Expected Output

job.batch/traefikee-bootstrap created

Note

Alternatively, you can use an environment variable, and set the license key without changing the file:

Command

export TRAEFIKEE_LICENSE_KEY="YOUR-LICENSE_KEY-VALUE"
curl -sSL https://s3.amazonaws.com/traefikee/examples/v1.2.1/kubernetes/multi-cn/hooks/bootstrap-job.yaml \
| sed "s/--licensekey=.*/--licensekey=${TRAEFIKEE_LICENSE_KEY}\"/g" \
| kubectl apply -f -

Connect traefikeectl to the New Cluster¶

About a minute after applying the bootstrap job, you should be able to connect to the cluster.

Configure traefikeectl to have access to the new cluster.

Command

traefikeectl connect --kubernetes

Expected Output

Retrieving TraefikEE Control credentials...ok
Removing cluster credentials from platform...ok
Credentials saved in "$HOME/.config/traefikee/traefikee", please make sure to keep them safe as they can never be retrieved again.
✔ Successfuly gained access to the cluster. You can now use other traefikeectl commands.

In case of an error

If traefikeectl connect displays the following error: unable to retrieve API credentials, it might be because the bootstrap job isn't done preparing your cluster yet. Wait a few seconds and try again.

One-time operation

When running traefikeectl connect, your credentials will be retrieved and it will not be possible to do it again in the future without re-installing a TraefikEE cluster. Remember to keep your credentials safe!

Check if the bootstrap node started correctly by listing the cluster nodes.

The bootstrap node's role should be CONTROL NODE, and its availability should be ACTIVE:

Command

traefikeectl list-nodes

Expected Output

Name                                            Availability  Role          Leader
----                                            ------------  ----          ------
bootstrap-traefikee-bootstrap-647965c6b5-xfktd  ACTIVE        CONTROL NODE  YES

Create the Control Nodes¶

Installing behind a proxy

In order to install a behind proxy, you must define the HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables for the control node. As the control node will make requests to the kubernetes API server, the NO_PROXY variable must contain its IP address. It can be found using kubectl get service kubernetes.

containers:
  - name: ...
    # [...]
    env:
      - name: HTTP_PROXY
        value: http://127.0.0.1:3129 # Your HTTP proxy URL
      - name: HTTPS_PROXY
        value: http://127.0.0.1:3129 # Your HTTPS proxy URL
      - name: NO_PROXY
        value: 100.64.0.1 # IP of the kubernetes service

Create the 3 control nodes for your cluster:

Command

kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.2.1/kubernetes/multi-cn/control-node-statefulset-multi-cn.yaml

Expected Output

statefulset.apps/traefikee-control-node created

Verify that you have 4 running pods for the 3 + 1 control nodes:

Command

kubectl get pod --namespace=traefikee --selector='app=traefikee' --selector='component=control-nodes'

Expected Output

NAME                                  READY     STATUS    RESTARTS   AGE
...
traefikee-control-node-0              1/1       Running   0          1m
traefikee-control-node-1              1/1       Running   0          47s
traefikee-control-node-2              1/1       Running   0          38s
...

Remove the Bootstrap Node¶

TraefikEE requires an odd number of control nodes to work flawlessly.

Remove the Bootstrap node to have only 3 control nodes (instead of 4):

Command

kubectl delete -f https://s3.amazonaws.com/traefikee/examples/v1.2.1/kubernetes/multi-cn/hooks/bootstrap-job.yaml

Expected Output

job.batch "traefikee-bootstrap" deleted

Verify that the pod of the Bootstrap node is completely deleted, so you only have 3 running pods for the 3 control nodes:

Command

kubectl get pod --namespace=traefikee --selector='app=traefikee' --selector='component=control-nodes'

Expected Output

NAME                       READY     STATUS    RESTARTS   AGE
traefikee-control-node-0   1/1       Running   0          2m
traefikee-control-node-1   1/1       Running   0          2m
traefikee-control-node-2   1/1       Running   0          2m

Every minute, the cluster cleans the node list and stops tracking the deleted nodes.

Validate that there are only 3 control nodes left after 1 minute:

Command

traefikeectl list-nodes

Expected Output

Name                        Availability  Role          Leader
----                        ------------  ----          ------
traefikee-control-node-1    ACTIVE        CONTROL NODE  YES
traefikee-control-node-2    ACTIVE        CONTROL NODE
traefikee-control-node-0    ACTIVE        CONTROL NODE

Important

The Bootstrap node was the previous leader and was removed. A new leader is required: an election is triggered to select a new leader. This behavior is what happens when a control node, acting as a leader, is going down in production: TraefikEE stays available.

Create Data Nodes¶

Create the data nodes, to handle your application traffic:

Command

kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.2.1/kubernetes/multi-cn/data-node-deployment.yaml

Expected Output

deployment.apps "traefikee-data-node" created

Verify that the 2 pods running the 2 data nodes are running:

Command

kubectl get pod --namespace=traefikee --selector='app=traefikee' --selector='component=data-nodes'

Expected Output

NAME                                   READY     STATUS    RESTARTS   AGE
traefikee-data-node-867cc84788-j78qg   1/1       Running   0          50s
traefikee-data-node-867cc84788-s9qwm   1/1       Running   0          50s

Validate that the TraefikEE cluster sees the 2 data nodes as members, with an availability "ACTIVE":

Command

traefikeectl list-nodes

Expected Output

Name                                            Availability  Role          Leader
----                                            ------------  ----          ------
traefikee-control-node-1                        ACTIVE        CONTROL NODE  YES
data-node-traefikee-data-node-68d856488c-5vx5n  ACTIVE        DATA NODE
data-node-traefikee-data-node-68d856488c-stgns  ACTIVE        DATA NODE
traefikee-control-node-2                        ACTIVE        CONTROL NODE
traefikee-control-node-0                        ACTIVE        CONTROL NODE

Backup your Installation¶

Don't forget to setup regular backups using the traefikeectl backup command. More information can be found in the backup and restore documentation.

What's Next?

Now that you have an up running cluster, you can configure your routing.