Skip to main content
Star us on GitHub Star

Install the Router in Kubernetes

ziti-router

Version: 1.0.6 Type: application AppVersion: 1.1.3

Host an OpenZiti router in Kubernetes

Add the OpenZiti Charts Repo to Helm

helm repo add openziti https://docs.openziti.io/helm-charts/

Minimal Installation

After adding the charts repo to Helm, then you may install the chart in the same cluster where the controller is running by using the cluster-internal service of the control plane endpoint. This default values used in this minimal approach is suitable for a Kubernetes distribution like K3S or Minikube that configures pass-through TLS for Service resources of type LoadBalancer.

# get a router enrollment token from the controller's management API
ziti edge create edge-router router1 \
--role-attributes default --tunneler-enabled --jwt-output-file /tmp/router1.jwt

# subscribe to the openziti Helm repo
helm repo add openziti https://openziti.github.io/helm-charts/

# install the router chart
helm install \
--namespace ziti-router --create-namespace --generate-name \
openziti/ziti-router \
--set-file enrollmentJwt=/tmp/router1.jwt \
--set advertisedHost=ziti-router.example.com \
--set ctrl.endpoint=ziti-controller-ctrl.ziti-controller.svc:6262

You must supply some values when you install the chart:

KeyTypeDefaultDescription
enrollmentJwtstringnilthe router enrollment token from the Ziti management API
advertisedHoststringnilthe DNS name that edge clients will resolve to reach this router's edge listener
ctrl.endpointstringnilthe DNS name:port of the router control plane endpoint provided by the Ziti controller

Managed Kubernetes Installation

Managed Kubernetes providers typically configure server TLS for a Service of type LoadBalancer. Ziti needs pass-through TLS because edge clients authenticate to the router with client certificates. We'll accomplish this by changing the Service type to ClusterIP and creating Ingress resources with pass-through TLS for each cluster service.

This example demonstrates creating TLS pass-through Ingress resources for use with ingress-nginx.

Ensure you have the ingress-nginx chart installed with controller.extraArgs.enable-ssl-passthrough=true. You can verify this feature is enabled by running kubectl describe pods {ingress-nginx-controller pod} and checking the args for --enable-ssl-passthrough=true.

If not enabled, then you must patch the ingress-nginx deployment to enable the SSL passthrough option.

kubectl patch deployment "ingress-nginx-controller" \
--namespace ingress-nginx \
--type json \
--patch '[{"op": "add",
"path": "/spec/template/spec/containers/0/args/-",
"value":"--enable-ssl-passthrough"
}]'
# subscribe to ingress-nginx
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx/

# install ingress-nginx
helm install \
--namespace ingress-nginx --create-namespace --generate-name \
ingress-nginx/ingress-nginx \
--set controller.extraArgs.enable-ssl-passthrough=true

Create a Helm chart values file for this router chart.

# /tmp/router-values.yml
ctrl:
endpoint: ziti-controller-ctrl.ziti-controller.svc:6262
advertisedHost: ziti-router.example.com
edge:
advertisedPort: 443
service:
type: ClusterIP
ingress:
enabled: true
ingressClassName: nginx
annotations:
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"

Now upgrade your router chart release with the values file.

# will attempt enrollment again if it failed initially
helm upgrade \
--namespace ziti-router ziti-router-123456789 \
openziti/ziti-router \
--set-file enrollmentJwt=/tmp/router1.jwt \
--values /tmp/router-values.yml

The minimal installation guided you to install a router in the same cluster as the controller, and the managed Kubernetes upgrade guided you to expose the router's edge listener as a pass-through TLS Ingress. Building on those concepts, let's expand your mesh of Ziti routers. For this you will need to configure router link listeners, i.e. router-to-router links. This is accomplished in this chart by setting some additional values.

Merge the following with your router values.

linkListeners:
transport:
advertisedHost: router1-transport.example.com
advertisedPort: 443
service:
enabled: true
type: ClusterIP
ingress:
enabled: true
ingressClassName: nginx
annotations:
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"

Notice that we've chosen a distinct DNS name for this new ingress. This allows us to have any number of 443/tcp virtual servers on the same IP address. You may find it convenient to delegate a DNS zone with a wildcard record resolving to your Nginx LoadBalancer IP.

Now upgrade your router chart release with the merged values file.

helm upgrade \
--namespace ziti-router ziti-router-123456789 \
openziti/ziti-router \
--set-file enrollmentJwt=/tmp/router1.jwt \
--values /tmp/router-values.yml

Proxy tunnel mode

The Openziti router supports the Proxy Tunnel mode. If you need to deploy Kubernetes services together with your Ziti Router in order to make these ports available as ClusterIP, NodePort or LoadBalancer services within your cluster, you can let this helm chart deploy those services for you. In some specific cases, it is not enough to have just one Kubernetes service making accessible all ports assigned to specific Openziti services, but to have some of these proxy ports on one, and others on another Kubernetes service (for example, if you want to expose one of the proxy services as a ClusterIP service, but another as a LoadBalancer service).

Here's an example router values' snippet to merge with your other values:

tunnel:
mode: proxy
proxyServices:
# this will be bound on the "default" proxy Kubernetes service, see below
- zitiService: my-ziti-service.svc
containerPort: 10443
advertisedPort: 10443
# this will be bound on an additionally configured proxy Kubernetes service, see below
- zitiService: my-other-service.svc
containerPort: 10022
advertisedPort: 10022
proxyDefaultK8sService:
enabled: true
type: ClusterIP
proxyAdditionalK8sServices:
- name: myservice
type: LoadBalancer
annotations:
metallb.universe.tf/loadBalancerIPs: 192.168.1.100

Values Reference

KeyTypeDefaultDescription
additionalVolumeslist[]additional volumes to mount to ziti-router container
advertisedHoststringnilcommon advertise-host for transport and edge listeners can also be specified separately via edge.advertisedHost and linkListeners.transport.advertisedHost
affinityobject{}deployment template spec affinity
configFilestring"ziti-router.yaml"filename of router config YAML
configMountDirstring"/etc/ziti/config"writeable mountpoint where read-only config file is projected to allow router to write ./endpoints statefile in same dir
csr.sans.dnslist[]additional DNS SANs
csr.sans.iplist[]additional IP SANs
ctrl.endpointstringnilrequired control plane endpoint
dnsConfigobject{}it allows to override dns options when dnsPolicy is set to None.
dnsPolicystring"ClusterFirstWithHostNet"
edge.advertisedHoststringnilDNS name that edge clients will use to reach this router's edge listener
edge.advertisedPortint443cluster service, node port, load balancer, and ingress port
edge.containerPortint3022cluster service target port on the container
edge.enabledbooltrueenable the edge listener in the router config
edge.ingress.annotationsstringnilingress annotations, e.g., to configure ingress-nginx
edge.ingress.enabledboolfalsecreate an ingress for the cluster service
edge.service.annotationsstringnilservice annotations
edge.service.enabledbooltruecreate a cluster service for the edge listener
edge.service.labelsstringnilservice labels
edge.service.typestring"ClusterIP"expose the service as a ClusterIP, NodePort, or LoadBalancer
enrollJwtFilestring"enrollment.jwt"
enrollmentJwtstringnilenrollment one time token from the controller's management API
envstringnilset name to value in containers' environment
execMountDirstring"/usr/local/bin"read-only mountpoint for executables (must be in image's executable search PATH)
fabric.metrics.enabledboolfalseconfigure fabric metrics in the router config
forwarder.latencyProbeIntervalint10
forwarder.linkDialQueueLengthint1000
forwarder.linkDialWorkerCountint32
forwarder.rateLimitedQueueLengthint5000
forwarder.rateLimitedWorkerCountint64
forwarder.xgressDialQueueLengthint1000
forwarder.xgressDialWorkerCountint128
hostNetworkboolfalseHost networking requested for a pod if set, i.e. tproxy ports enabled in the host namespace. i.e. egress gateway
identityMountDirstring"/etc/ziti/identity"read-only mountpoint for router identity secret specified in deployment for use by router run container
image.additionalArgslist[]additional arguments can be passed directly to the container to modify ziti runtime arguments
image.argslist["run","{{ .Values.configMountDir }}/{{ .Values.configFile }}"]deployment container command args and opts
image.commandlist["/entrypoint.bash"]deployment container command
image.pullPolicystring"Always"deployment image pull policy
image.repositorystring"docker.io/openziti/ziti-router"container image tag for deployment
image.tagstringnilcontainer image tag (default is Chart's appVersion)
linkListeners.transport.advertisedHoststringnilDNS name that other routers will use to form mesh transport links with this router. Default is cluster-internal service DNS name:port.
linkListeners.transport.advertisedPortint443cluster service, node port, load balancer, and ingress port
linkListeners.transport.containerPortint10080cluster service target port on the container
linkListeners.transport.ingress.annotationsstringnilingress annotations, e.g., to configure ingress-nginx
linkListeners.transport.ingress.enabledboolfalsecreate an ingress for the cluster service
linkListeners.transport.service.annotationsstringnilservice annotations
linkListeners.transport.service.enabledbooltruecreate a cluster service for the router transport link listener
linkListeners.transport.service.labelsstringnilservice labels
linkListeners.transport.service.typestring"ClusterIP"expose the service as a ClusterIP, NodePort, or LoadBalancer
nodeSelectorobject{}deployment template spec node selector
persistence.accessModestring"ReadWriteOnce"PVC access mode: ReadWriteOnce (concurrent mounts not allowed), ReadWriteMany (concurrent allowed)
persistence.annotationsobject{}annotations for the PVC
persistence.enabledbooltruerequired: place a storage claim for the ctrl endpoints state file
persistence.existingClaimstring""A manually managed Persistent Volume and Claim Requires persistence.enabled: true If defined, PVC must be created manually before volume will be bound
persistence.sizestring"50Mi"50Mi is plenty for this state file
persistence.storageClassstring""Storage class of PV to bind. By default it looks for the default storage class. If the PV uses a different storage class, specify that here.
persistence.volumeNamestringnilPVC volume name
podAnnotationsobject{}annotations to apply to all pods deployed by this chart
podSecurityContextobject{"fsGroup":2171}deployment template spec security context
podSecurityContext.fsGroupint2171this is the GID of "ziggy" run-as user in the container that has access to any files created by the router process in the emptyDir volume used to persist the endpoints state file
proxyobject{}Explicit proxy setting in the router configuration. Router can be deployed in a site where all egress traffic is forwarded through an explicit proxy. The enrollment will also be forwarded through the proxy.
resourcesobject{}deployment container resources
securityContextstringnildeployment container security context
tolerationslist[]deployment template spec tolerations
tunnel.diverterPathstringnilthe tproxy mode can be switched from iptables based interception to bpf interception by passing the user space bpf program path. bpf kernel space program is expected to be loaded prior or during router deployment, e.g. bpfman agent, hostpath, etc
tunnel.dnsSvcIpRangestringnilCIDR range for the internal service fqdn to dynamic intercept IP address resolution (default: 100.64.0.0/10)
tunnel.lanIfstring"lo"interface device name for setting up INPUT firewall rules if fw enabled. It must be set but not needed in containers. Thus, it is set to lo by default
tunnel.modestring"none"run mode for the router's built-in tunnel component: host, tproxy, proxy, or none
tunnel.proxyAdditionalK8sServiceslist[]if tunnel mode is "proxy", create a separate cluster service for each Ziti service listed in "proxyServices" which k8sService == name
tunnel.proxyDefaultK8sServiceobject{"enabled":true,"type":"ClusterIP"}if tunnel mode is "proxy", create the a cluster service named {{ release }}-proxy-default listening on each "advertisedPort" defined in "proxyServices"
tunnel.proxyServiceslist[]list of Ziti services for which K8s services are to be created by this deployment, default is one cluster service port per Ziti service
tunnel.resolverstringnilZiti nameserver listener where OS must be configured to send DNS queries (default: udp://127.0.0.1:53)

TODO's

  • replicas - does it make sense? afaik every replica needs it's own identity - how does this fit in?
  • lower CA / Cert lifetime; refresh certificates on update