Skip to content

Advanced Installation of Traefik Enterprise Edition with Multiple Control Nodes on Kubernetes with YAML

This installation guide is for experts who want to fine-tune their TraefikEE (Traefik Enterprise Edition) installation.

It covers how to install TraefikEE using Kubernetes YAML files as a Kubernetes Ingress Controller.

Kubernetes Knowledge

Assistance with configuring or setting up a Kubernetes cluster is not included in this guide. If you need more information about Kubernetes, start with the following resources:

Requirements

  • The traefikeectl tool installed

  • A Kubernetes cluster:

    • Supported versions: 1.10, 1.11, 1.12 and 1.13
    • RBAC enabled (recommended),
    • Access to the Kubernetes API with kubectl, with the ability to create and manage namespaces and their resources.

Create the Namespace

Create a namespace named traefikee to host the TraefikEE installation:

kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.0.0/kubernetes/multi-cn/namespace.yaml
namespace/traefikee created

Prepare Role Based Access Control (RBAC)

Create the RBAC objects to allow TraefikEE pods to interact with the Kubernetes API:

kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.0.0/kubernetes/multi-cn/rbac.yaml
serviceaccount/traefikee created
clusterrole.rbac.authorization.k8s.io/traefikee-role created
clusterrolebinding.rbac.authorization.k8s.io/traefikee created
serviceaccount/traefikee-bootstrap created
clusterrole.rbac.authorization.k8s.io/traefikee-bootstrap-role created
clusterrolebinding.rbac.authorization.k8s.io/traefikee-bootstrap created
Note

The following Service Accounts are created:

  • serviceaccount/traefikee, authorized to access the Kubernetes API actions required for TraefikEE.
  • serviceaccount/traefikee-bootstrap, authorized to manage secrets for TraefikEE installation.

Create the Service

Create the services to allow network access to the TraefikEE cluster:

kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.0.0/kubernetes/multi-cn/data-node-external-service.yaml
kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.0.0/kubernetes/multi-cn/control-node-internal-service.yaml
kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.0.0/kubernetes/multi-cn/cluster-internal-api-service.yaml
service "traefikee-lb" created
service "traefikee-control-nodes" created
service "traefikee-api" created
Note

The following services are created:

  • traefikee-lb handles the traffic of the ingress rules for your applications.
  • traefikee-control-nodes handles the traffic for control nodes internal communication.
  • traefikee-api handles the traffic for TraefikEE's API internal usage such as metric collection or Web UI.

Create the Bootstrap Node

Installing behind a proxy

In order to install a behind proxy, you must define the HTTP_PROXY and HTTPS_PROXY environment variables for all TraefikEE containers. You will need to edit each container ressource definition in this documentation and add the following snippet.

containers:
  - name: ...
    # [...]
    env:
      - name: HTTP_PROXY
        value: http://127.0.0.1:3129 # Your HTTP proxy URL
      - name: HTTPS_PROXY
        value: http://127.0.0.1:3129 # Your HTTPS proxy URL

The "Bootstrap node" is an ephemeral control node, only responsible for initializing the cluster with your license information.

Download the file bootstrap-job.yaml:

curl -sSLO \
  https://s3.amazonaws.com/traefikee/examples/v1.0.0/kubernetes/multi-cn/hooks/bootstrap-job.yaml
  • Insert your license key in the file bootstrap-job.yaml:
    • Search for line --licensekey=fakeLicense
    • Replace the value fakeLicense with your license key
Bootstrap stopped after 5 minutes

If the control plane is not initialized within 5 minutes, then the bootstrap node will stop automatically. To avoid this, you have to tune both following parameters. For example, let's wait 600 seconds:

  • activeDeadlineSeconds: 600 (in the section spec)
  • --timeout=600 (in the section spec/templates/spec/containers/args)
  • Save the file

Create the Bootstrap node to initialize the cluster:

kubectl apply -f ./bootstrap-job.yaml
job.batch/traefikee-bootstrap created
Note

Alternatively, you can use an environment variable, and set the license key without changing the file:

export TRAEFIKEE_LICENSE_KEY="YOUR-LICENSE_KEY-VALUE"
curl -sSL https://s3.amazonaws.com/traefikee/examples/v1.0.0/kubernetes/multi-cn/hooks/bootstrap-job.yaml \
| sed "s/--licensekey=.*/--licensekey=${TRAEFIKEE_LICENSE_KEY}\"/g" \
| kubectl apply -f -

Connect traefikeectl to the New Cluster

About a minute after applying the bootstrap job, you should be able to connect to the cluster.

Configure traefikeectl to have access to the new cluster.

traefikeectl connect --kubernetes
Retrieving TraefikEE Control credentials...ok
Removing cluster credentials from platform...ok
Credentials saved in "$HOME/.config/traefikee/traefikee", please make sure to keep them safe as they can never be retrieved again.
✔ Successfuly gained access to the cluster. You can now use other traefikeectl commands.
In case of an error

If traefikeectl connect displays the following error: unable to retrieve API credentials, it might be because the bootstrap job isn't done preparing your cluster yet. Wait a few seconds and try again.

One-time operation

When running traefikeectl connect, your credentials will be retrieved and it will not be possible to do it again in the future without re-installing a TraefikEE cluster. Remember to keep your credentials safe!

Check if the bootstrap node started correctly by listing the cluster nodes.

The bootstrap node's role should be CONTROL NODE, and its availability should be ACTIVE:

traefikeectl list-nodes
Name                                            Availability  Role          Leader
----                                            ------------  ----          ------
bootstrap-traefikee-bootstrap-647965c6b5-xfktd  ACTIVE        CONTROL NODE  YES

Create Token Secrets

Get the data node token generated by the bootstrap node, and put it into the environment variable DATA_NODE_TOKEN:

export DATA_NODE_TOKEN="$(kubectl exec -t \
  --namespace=traefikee \
  $(kubectl get pods --namespace traefikee --selector=app=traefikee,component=control-nodes --output jsonpath="{.items[*].metadata.name}") \
  -- /traefikee env | grep 'DATA_NODE' | cut -d '"' -f2)"

Do the same for the control node token, and put it into the environment variable CONTROL_NODE_TOKEN:

export CONTROL_NODE_TOKEN="$(kubectl exec -t \
  --namespace=traefikee \
  $(kubectl get pods --namespace traefikee --selector=app=traefikee,component=control-nodes --output jsonpath="{.items[*].metadata.name}") \
  -- /traefikee env | grep 'CONTROL_NODE' | cut -d '"' -f2)"

Validate that the variables are set properly:

env | grep NODE_TOKEN
CONTROL_NODE_TOKEN=...6645376267354e51734d59336a457369712...
DATA_NODE_TOKEN=...6645376267354e51734d59336a457369712...

Download the file cluster-secret.yaml:

curl -sSLO \
  https://s3.amazonaws.com/traefikee/examples/v1.0.0/kubernetes/multi-cn/hooks/cluster-secret.yaml

From the content of the file cluster-secret.yaml, insert the values of the token, and apply the modified content to Kubernetes:

cat cluster-secret.yaml \
  | sed "s/data-node:.*$/data-node: $DATA_NODE_TOKEN/" \
  | sed "s/control-node:.*$/control-node: $CONTROL_NODE_TOKEN/" \
  | kubectl apply -f -
secret/traefikee-tokens configured

Verify the secret named traefikee-tokens, of type "Opaque" exists in Kubernetes:

kubectl get secrets --namespace=traefikee
NAME                 TYPE      DATA      AGE
...
traefikee-tokens     Opaque    2         52s
...
Note

You can retrieve the base64 encoded values stored in Kubernetes with the following commands:

  • Control node token:
kubectl get secret --namespace=traefikee \
    traefikee-tokens --output="jsonpath={.data.control-node}"
  • Data node token:
kubectl get secret --namespace=traefikee \
    traefikee-tokens --output="jsonpath={.data.data-node}"

Allow TraefikEE To Generate Token Secrets Automatically

When the TraefikEE bootstrap node is run in kubernetes, it will automatically attempt to connect to the kubernetes API, and generate the token secret without requiring user interaction. It will create the secret in the same namespace as the bootstrap with the same traefikee-tokens name as in the guide above.

You may not have to manually generate these tokens and this secret if your RBAC for the bootstrap node allows secret creation.

Note

By allowing the bootstrap node to create this secret, you can reduce the number of steps needed to bootstrap your cluster.

Create the Control Nodes

Create the 3 control nodes for your cluster:

kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.0.0/kubernetes/multi-cn/control-node-statefulset.yaml
statefulset.apps/traefikee-control-node created

Verify that you have 4 running pods for the 3 + 1 control nodes:

kubectl get pod --namespace=traefikee --selector='app=traefikee' --selector='component=control-nodes'
NAME                                  READY     STATUS    RESTARTS   AGE
traefikee-bootstrap-b7d57f77d-7w8lq   1/1       Running   0          21m
traefikee-control-node-0              1/1       Running   0          1m
traefikee-control-node-1              1/1       Running   0          47s
traefikee-control-node-2              1/1       Running   0          38s

Remove the Bootstrap Node

TraefikEE requires an odd number of control nodes to work flawlessly.

Remove the Bootstrap node to have only 3 control nodes (instead of 4):

kubectl delete -f https://s3.amazonaws.com/traefikee/examples/v1.0.0/kubernetes/multi-cn/hooks/bootstrap-job.yaml
job.batch "traefikee-bootstrap" deleted

Verify that the pod of the Bootstrap node is completely deleted, so you only have 3 running pods for the 3 control nodes:

kubectl get pod --namespace=traefikee --selector='app=traefikee' --selector='component=control-nodes'
NAME                       READY     STATUS    RESTARTS   AGE
traefikee-control-node-0   1/1       Running   0          2m
traefikee-control-node-1   1/1       Running   0          2m
traefikee-control-node-2   1/1       Running   0          2m

Every minute, the cluster cleans the node list and stops tracking the deleted nodes.

Validate that there are only 3 control nodes left after 1 minute:

traefikeectl list-nodes
Name                        Availability  Role          Leader
----                        ------------  ----          ------
traefikee-control-node-1    ACTIVE        CONTROL NODE  YES
traefikee-control-node-2    ACTIVE        CONTROL NODE
traefikee-control-node-0    ACTIVE        CONTROL NODE

Important

The Bootstrap node was the previous leader and was removed. A new leader is required: an election is triggered to select a new leader. This behavior is what happens when a control node, acting as a leader, is going down in production: TraefikEE stays available.

Create Data Nodes

Create the data nodes, to handle your application traffic:

kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.0.0/kubernetes/multi-cn/data-node-deployment.yaml
deployment.apps "traefikee-data-node" created

Verify that the 2 pods running the 2 data nodes are running:

kubectl get pod --namespace=traefikee --selector='app=traefikee' --selector='component=data-nodes'
NAME                                   READY     STATUS    RESTARTS   AGE
traefikee-data-node-867cc84788-j78qg   1/1       Running   0          50s
traefikee-data-node-867cc84788-s9qwm   1/1       Running   0          50s

Validate that the TraefikEE cluster sees the 2 data nodes as members, with an availability "ACTIVE":

traefikeectl list-nodes
Name                                            Availability  Role          Leader
----                                            ------------  ----          ------
traefikee-control-node-1                        ACTIVE        CONTROL NODE  YES
data-node-traefikee-data-node-68d856488c-5vx5n  ACTIVE        DATA NODE
data-node-traefikee-data-node-68d856488c-stgns  ACTIVE        DATA NODE
traefikee-control-node-2                        ACTIVE        CONTROL NODE
traefikee-control-node-0                        ACTIVE        CONTROL NODE

Backup your Installation

Don't forget to setup regular backups using the traefikeectl backup command. More information can be found in the backup and restore documentation.