VMware Tanzu¶
Overview of VMware Tanzu¶
VMware Tanzu is a suite of products that helps users run and manage multiple Kubernetes (K8S) clusters across public and private “clouds”. Tanzu Kubernetes Grid (TKG) is a multi-cloud Kubernetes footprint that you can run both on-premises in vSphere and in the public cloud on Amazon EC2 and Microsoft Azure. TKG by default uses Antrea CNI.
What is NodePortLocal?¶
For clusters using Antrea as CNI, the NodePortLocal feature option can be used; a Pod can be directly reached from an external network through a port in the Node. In this mode, instead of relying on NodePort Services implemented by kube-proxy, CIS consumes NPL port mappings published by the Antrea Agent (as K8s Pod annotations) to read NodePort and NodeIP information to load balance traffic to backend pods.
Configuring VMware Tanzu¶
Prerequisites¶
- Deploy Tanzu Kubernetes cluster with Antrea CNI. See VMware documentation for more information.
- Enable nodeportLocal in antrea-agent config. Prior to Antrea version 1.4, a feature gate, NodePortLocal, must be enabled on the antrea-agent for the feature to work. From version 1.4, it is enabled by default.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | kind: ConfigMap apiVersion: v1 metadata: name: antrea-config-dcfb6k2hkm namespace: kube-system data: antrea-agent.conf: | featureGates: # True by default starting with Antrea v1.4 # NodePortLocal: true nodePortLocal: enable: true # Uncomment if you need to change the port range. # portRange: 61000-62000 |
Configuration¶
To enable NPL feature, set
--pool-member-type
tonodeportlocal
in CIS arguments. This is only applicable for Antrea CNI enabled clusters.args: --pool-member-type="nodeportlocal"
Example:
cis-deploy.yaml¶1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
apiVersion: apps/v1 kind: Deployment metadata: name: k8s-bigip-ctlr-deployment namespace: kube-system spec: # DO NOT INCREASE REPLICA COUNT replicas: 1 selector: matchLabels: app: k8s-bigip-ctlr-deployment template: metadata: labels: app: k8s-bigip-ctlr-deployment spec: # Name of the Service Account bound to a Cluster Role with the required # permissions containers: - name: k8s-bigip-ctlr image: "f5networks/k8s-bigip-ctlr:latest" env: - name: BIGIP_USERNAME valueFrom: secretKeyRef: # Replace with the name of the Secret containing your login # credentials name: bigip-login key: username - name: BIGIP_PASSWORD valueFrom: secretKeyRef: # Replace with the name of the Secret containing your login # credentials name: bigip-login key: password command: ["/app/bin/k8s-bigip-ctlr"] args: [ # See the k8s-bigip-ctlr documentation for information about # all config options # https://clouddocs.f5.com/containers/latest/ "--bigip-username=$(BIGIP_USERNAME)", "--bigip-password=$(BIGIP_PASSWORD)", "--bigip-url=<ip_address-or-hostname>", "--bigip-partition=<name_of_partition>", "--pool-member-type=nodeportlocal", "--insecure", ] serviceAccountName: bigip-ctlr
Push this configuration with the following command:
kubectl apply -f cis_deploy.yaml
Services used should be annotated with
nodeportlocal.antrea.io/enabled: "true"
for selecting pods for NodePortLocal. This enablesNodePortLocal
for all the Pods which are selected by the Service through a selector, and the ports of these Pods will be reachable through Node ports allocated. The selected Pods will be annotated with the details about allocated Node port for the Pod.service.yaml¶1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44
apiVersion: v1 kind: Service metadata: annotations: nodeportlocal.antrea.io/enabled: "true" labels: app: f5-hello-world name: f5-hello-world spec: ports: - name: f5-hello-world port: 8080 protocol: TCP targetPort: 8080 selector: app: f5-hello-world type: ClusterIP --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: f5-hello-world name: f5-hello-world spec: replicas: 2 selector: matchLabels: app: f5-hello-world template: metadata: labels: app: f5-hello-world spec: containers: - env: - name: service_name value: f5-hello-world image: f5devcentral/f5-hello-world:latest imagePullPolicy: Always name: f5-hello-world ports: - containerPort: 8080 protocol: TCP
If the NodePortLocal feature gate is enabled, then all the Pods in the Deployment will be annotated with the
nodeportlocal.antrea.io
annotation. The value of this annotation is a serialized JSON array. For the example above, the pod may look like this:kubectl describe po f5-hello-world-6d859874b7-prf8l -n cis Name: f5-hello-world-6d859874b7-prf8l Namespace: cis Priority: 0 Start Time: Wed, 09 Mar 2022 00:17:30 -0800 Labels: app=f5-hello-world pod-template-hash=6d859874b7 Annotations: kubernetes.io/psp: cis-psp nodeportlocal.antrea.io: [{"podPort":8080,"nodeIP":"10.244.0.3","nodePort":40001}]
This annotation indicates that port 8080 of the Pod can be reached through port 40001 of the Node with IP Address 10.244.0.3. The
nodeportlocal.antrea.io
annotation is created and managed by Antrea.Note
NodePortLocal can only be used with Services of type
ClusterIP
. Thenodeportlocal.antrea.io
annotation has no effect for Services of typeNodePort
orExternalName
. The annotation also has no effect for Services with an empty or missing Selector.
Limitations¶
CIS currently supports the NPL feature with Ingress, ConfigMap, and virtualserver resource. The feature is validated on k8s Tanzu infrastructure.
Pod Security Policies with Tanzu Kubernetes Clusters¶
Tanzu Kubernetes clusters are provisioned with the PodSecurityPolicy Admission Controller enabled. So, pod security policy is required to deploy workloads. Cluster administrators can deploy pods from their user account to any namespace, and from service accounts to the kube-system namespace. For all other use cases, you need to explicitly bind to a PodSecurityPolicy object.
For example, if you create a deployment without PSP, pod creation fails and you may observe below in namespace events.
kubectl get events -n test
LAST SEEN TYPE REASON OBJECT MESSAGE
19s Warning FailedCreate replicaset/f5-hello-world-7bb546899 Error creating: pods "f5-hello-world-7bb546899-" is forbidden: PodSecurityPolicy: unable to admit pod: []
To resolve this, create a PSP and a Cluster Role that grant access to use the desired policies by following the steps below.
Create a PodSecurityPolicy.
kubectl create -f psp.yaml -n test
sample-psp.yaml¶1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: test-psp spec: privileged: false # Prevents creation of privileged Pods supplementalGroups: rule: RunAsAny runAsUser: rule: RunAsAny fsGroup: rule: RunAsAny seLinux: rule: RunAsAny volumes: - '*'
Create a cluster role to grant access to the policy.
kubectl create -f clusterrole.yaml
clusterrole.yaml¶1 2 3 4 5 6 7 8 9 10 11 12 13
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: test-clusterrole rules: - apiGroups: - policy resources: - podsecuritypolicies verbs: - use resourceNames: - test-psp
Create rolebinding for namespace access.
kubectl create -f rolebinding.yaml
rolebinding.yaml¶1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
# Bind the ClusterRole to the desired set of service accounts. # Policies should typically be bound to service accounts in a namespace. apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: test-rolebinding namespace: test roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: test-clusterrole subjects: # Example: All service accounts in my-namespace - apiGroup: rbac.authorization.k8s.io kind: Group name: system:serviceaccounts:test
Examples Repository¶
Note
To provide feedback on Container Ingress Services or this documentation, please file a GitHub Issue.