Skip to content

Advanced Installation of Traefik Enterprise Edition on Kubernetes with YAML

This installation guide is for experts who want to fine tune their TraefikEE (Traefik Enterprise Edition) installation.

It covers how to install TraefikEE using Kubernetes YAML files as a Kubernetes Ingress Controller.

Kubernetes Knowledge

Assistance with configuring or setting up a Kubernetes cluster are not included in this guide. If you need more information about Kubernetes, start with the following resources:

Requirements

  • The traefikeectl tool installed

  • A Kubernetes cluster:

    • Supported versions: 1.10, 1.11, 1.12 and 1.13
    • RBAC enabled (recommended),
    • Access to the Kubernetes API with kubectl, with the ability to create and manage namespaces and their resources.

Create the Namespace

Create a namespace named traefikee to host the TraefikEE's installation:

kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.0.0-beta15/kubernetes/namespace.yaml
namespace/traefikee created

Prepare Role Based Access Control (RBAC)

Create the RBAC objects to allow TraefikEE's pods to interact with the Kubernetes API:

kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.0.0-beta15/kubernetes/rbac.yaml
serviceaccount/traefikee created
clusterrole.rbac.authorization.k8s.io/traefikee-role created
clusterrolebinding.rbac.authorization.k8s.io/traefikee created
serviceaccount/traefikee-bootstrap created
clusterrole.rbac.authorization.k8s.io/traefikee-bootstrap-role created
clusterrolebinding.rbac.authorization.k8s.io/traefikee-bootstrap created
Note

The following Service Accounts are created:

  • serviceaccount/traefikee, authorized to access the Kubernetes API actions required for TraefikEE.
  • serviceaccount/traefikee-bootstrap, authorized to manage secrets for TraefikEE installation.

Create the Service

Create the services to allow network access to the TraefikEE's cluster:

kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.0.0-beta15/kubernetes/data-node-external-service.yaml
kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.0.0-beta15/kubernetes/control-node-internal-service.yaml
kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.0.0-beta15/kubernetes/cluster-internal-api-service.yaml
service "traefikee-lb" created
service "traefikee-control-nodes" created
service "traefikee-api" created
Note

The following services are created:

  • traefikee-lb handles the traffic of the ingress rules for your applications.
  • traefikee-control-nodes handles the traffic for control nodes internal communication.
  • traefikee-api handles the traffic for TraefikEE's API internal usage such as metric collection or Web UI.

Create the Bootstrap Node

The "Bootstrap node" is an ephemeral control node, only responsible for initializing the cluster with your license information.

Download the file bootstrap-deployment.yaml:

curl -sSLO \
  https://s3.amazonaws.com/traefikee/examples/v1.0.0-beta15/kubernetes/hooks/bootstrap-job.yaml
  • Insert your license key in the file bootstrap-job.yaml:
    • Search for line --licensekey=fakeLicense
    • Replace the value fakeLicense with your license key
Bootstrap stopped after 5 minutes

If the control plane is not initialized within 5 minutes, then the bootstrap node will stop automatically. To avoid this, you can tune the 2 following parameters. For example, let's wait 600 seconds:

  • activeDeadlineSeconds: 600 (in the section spec)
  • --timeout=120 (in the section spec/templates/spec/containers/args)
  • Save the file

Create the Bootstrap node to initialize the cluster:

kubectl apply -f ./bootstrap-job.yaml
job.batch/traefikee-bootstrap created
Note

Alternatively, you can use an environment variable, and set the license key without changing the file:

export TRAEFIKEE_LICENSE_KEY="YOU-LICENSE_KEY-VALUE"
curl -sSL https://s3.amazonaws.com/traefikee/examples/v1.0.0-beta15/kubernetes/hooks/bootstrap-job.yaml \
| sed "s/--licensekey=.*/--licensekey=${TRAEFIKEE_LICENSE_KEY}\"/g" \
| kubectl apply -f -

Check if the bootstrap node started correctly by listing the cluster nodes.

The bootstrap node's role should be Control Node (Current Leader), and its status should be READY:

traefikeectl list-nodes --kubernetes
Name           Role
----           --------------
bootstrap      Control Node (Current Leader)

Create Token's Secrets

Get the control node's token generated by the bootstrap node, and put it into the environment variable CONTROL_NODE_TOKEN:

export CONTROL_NODE_TOKEN="$(kubectl exec -t \
  --namespace=traefikee \
  $(kubectl get pods --namespace traefikee --selector=app=traefikee,component=control-nodes --output jsonpath="{.items[*].metadata.name}") \
  -- /traefikee env | grep 'CONTROL_NODE' | cut -d '"' -f2)"

Repeat the same for the data node's token with the environment variable DATA_NODE_TOKEN:

export DATA_NODE_TOKEN="$(kubectl exec -t \
  --namespace=traefikee \
  $(kubectl get pods --namespace traefikee --selector=app=traefikee,component=control-nodes --output jsonpath="{.items[*].metadata.name}") \
  -- /traefikee env | grep 'DATA_NODE' | cut -d '"' -f2)"

Validate that both variables are filed with the 2 different tokens:

env | grep NODE_TOKEN
CONTROL_NODE_TOKEN=...6332347a626e567a4e446c335932466961584e794d32...
DATA_NODE_TOKEN=...6645376267354e51734d59336a457369712...

Download the file cluster-secret.yaml:

curl -sSLO \
  https://s3.amazonaws.com/traefikee/examples/v1.0.0-beta15/kubernetes/hooks/cluster-secret.yaml

From the content of the file cluster-secret.yaml, insert the values of the 2 tokens, and apply the modified content to Kubernetes:

cat cluster-secret.yaml \
  | sed "s/control-node:.*$/control-node: $CONTROL_NODE_TOKEN/" \
  | sed "s/data-node:.*$/data-node: $DATA_NODE_TOKEN/" \
  | kubectl apply -f -
secret/traefikee-tokens configured

Verify the secret named traefikee-tokens, of type "Opaque" exists in Kubernetes:

kubectl get secrets --namespace=traefikee
NAME                 TYPE      DATA      AGE
...
traefikee-tokens     Opaque    2         52s
...
Note

You can retrieve the base64 encoded values stored in Kubernetes with the following commands:

  • Control node's token:
kubectl get secret --namespace=traefikee \
    traefikee-tokens --output="jsonpath={.data.control-node}"
  • Data node's token:
kubectl get secret --namespace=traefikee \
    traefikee-tokens --output="jsonpath={.data.data-node}"

Create the Control Nodes

Create the 3 control nodes for your cluster:

kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.0.0-beta15/kubernetes/control-node-statefulset.yaml
statefulset.apps/traefikee-control-node created

Verify that you have 4 running pods for the 3 + 1 control nodes:

kubectl get pod --namespace=traefikee --selector='app=traefikee' --selector='component=control-nodes'
NAME                                  READY     STATUS    RESTARTS   AGE
traefikee-bootstrap-b7d57f77d-7w8lq   1/1       Running   0          21m
traefikee-control-node-0              1/1       Running   0          1m
traefikee-control-node-1              1/1       Running   0          47s
traefikee-control-node-2              1/1       Running   0          38s

Remove the Bootstrap Node

TraefikEE requires an odd number of control nodes to work flawlessly.

Remove the Bootstrap node to have only 3 control nodes (instead of 4):

kubectl delete -f https://s3.amazonaws.com/traefikee/examples/v1.0.0-beta15/kubernetes/hooks/bootstrap-job.yaml
job.batch "traefikee-bootstrap" deleted

Verify that the pod of the Bootstrap node is completely deleted, so you only have 3 running pods for the 3 control nodes:

kubectl get pod --namespace=traefikee --selector='app=traefikee' --selector='component=control-nodes'
NAME                       READY     STATUS    RESTARTS   AGE
traefikee-control-node-0   1/1       Running   0          2m
traefikee-control-node-1   1/1       Running   0          2m
traefikee-control-node-2   1/1       Running   0          2m

Every minute, the cluster cleans the node list and stops tracking the deleted nodes.

Validate that there are only 3 control nodes left after 1 minute:

traefikeectl list-nodes --kubernetes
Name                      Role
----                      --------------
traefikee-control-node-1  Control Node (Current Leader)
traefikee-control-node-0  Control Node
traefikee-control-node-2  Control Node

Important

The Bootstrap node was the previous leader and was removed. A new leader is required: an election is triggered to select a new leader. This behavior is what happens when a control node, acting as a leader, is going down in production: TraefikEE stays available.

Create Data Nodes

Create the data nodes, to handle your application traffic:

kubectl apply -f https://s3.amazonaws.com/traefikee/examples/v1.0.0-beta15/kubernetes/data-node-deployment.yaml
deployment.apps "traefikee-data-node" created

Verify that the 2 pods running the 2 data nodes are running:

kubectl get pod --namespace=traefikee --selector='app=traefikee' --selector='component=data-nodes'
NAME                                   READY     STATUS    RESTARTS   AGE
traefikee-data-node-867cc84788-j78qg   1/1       Running   0          50s
traefikee-data-node-867cc84788-s9qwm   1/1       Running   0          50s

Validate that the TraefikEE cluster sees the 2 data nodes as members, with a status "READY":

traefikeectl list-nodes --kubernetes
Name                                  Role
----                                  --------------
traefikee-control-node-1              Control Node
traefikee-control-node-0              Control Node (Current Leader)
traefikee-control-node-2              Control Node
traefikee-data-node-867cc84788-j78qg  Data Node
traefikee-data-node-867cc84788-s9qwm  Data Node