Upgrading dSSM¶
The Service Proxy for Kubernetes (SPK) distributed Session State Management (dSSM) Sentinel and DB Pods can be upgraded using the typical Helm upgrade process. However, to ensure the process completes without service interruption, a custom dssm-upgrade-hook container is deployed during the upgrade, and requires additional permissions to complete the upgrade tasks. The upgrade process maintains all of the dSSM DB Pod session state data.
Note: If preserving data is not required, refer to the Quick Upgrade section to properly uninstall and upgrade the dSSM installation.
This document guides you through upgrading the dSSM database, and verifying the results.
Requirements¶
Ensure you have:
A running SPK dSSM Database installation.
Uploaded the f5-dssm-upgrader image with the SPK Software installation.
A newer version of the SPK dSSM Helm chart.
A workstation with Helm installed.
Procedures¶
Use the procedures below to upgrade the dSSM database, verify the results, and if required, rollback to the previous installation version.
Pre-upgrade status¶
Use the step below to verify the dSSM Pod cluster status, software version and persisted data. This will be useful to ensure the upgrade is succesful.
Ensure the dSSM installation Project is selected.
In this example, the dSSM Pods are in the spk-utilities Project:
oc project spk-utilities
Verify the STATUS of the dSSM Pods is Running.
oc get pods
NAME READY STATUS RESTARTS f5-dssm-db-0 2/2 Running 0 f5-dssm-db-1 2/2 Running 0 f5-dssm-db-2 2/2 Running 0 f5-dssm-sentinel-0 2/2 Running 0 f5-dssm-sentinel-1 2/2 Running 0 f5-dssm-sentinel-2 2/2 Running 0
Verify the f5-dssm-store version.
oc describe pods | grep Image: | grep -i dssm
In this example, the f5-dssm-store is v1.6.1:
Image: artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.6.1 Image: artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.6.1 Image: artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.6.1 Image: artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.6.1 Image: artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.6.1 Image: artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.6.1
Log in to the dSSM database (DB).
oc exec -it f5-dssm-db-0 -- bash
Enter the Redis command line interface (CLI).
redis-cli --tls --cert /tls/dssm/mds/svr/tls.crt \ --key /tls/dssm/mds/svr/tls.key \ --cacert /tls/dssm/mds/svr/ca.crt
List the DB entries. The entries should be present after the upgrade.
KEYS *1) "0073c3b6eft_dns4610.144.175.221" 2) "0073c3b6eft_dns4610.144.175.222" 3) "0073c3b6eft_dns4610.144.175.224" 4) "0073c3b6eft_dns4610.144.175.223" 5) "0073c3b6eft_dns4610.144.175.220"
Software upgrade¶
Use the steps below to upgrade the dSSM Sentinel and DB Pods.
Note: The dssm-upgrade-hook container logs valuable diagnostic data, opening a second shell to view the data is recommended.
Ensure the dSSM installation Project is selected.
oc project <name>
In this example, the dSSM Pods are in the spk-utilities Project:
oc project spk-utilities
The f5-dssm-upgrader image is provided with the SPK Software, and must be referenced using the dssm-values.yaml file below.
Note: Replace the local.registry.com value with the domain name of the local image registry.
dssmUpgrader: image: repository: "local.registry.com"
To grant the dssm-upgrade-hook container access the K8S API, create two YAML files with the following code, and set the
namespaceparameter to the dSSM installation Project:
Important: The dssm-upgrade-hook will fail to complete the upgrade without proper access to the K8S API.kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: pods-list namespace: spk-utilities rules: - apiGroups: ["", "apps"] resources: ["pods", "statefulsets", "statefulset"] verbs: ["get", "delete", "list", "watch"]
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: pods-list subjects: - kind: ServiceAccount name: f5-dssm namespace: spk-utilities roleRef: kind: Role name: pods-list apiGroup: rbac.authorization.k8s.io
Create the
RoleandRoleBindingobjects.oc create -f role.yaml
oc create -f role-binding.yaml
Verify the
RoleandRoleBindingobjects have been created.oc describe -f role.yaml
Name: pods-list Labels: <none> Annotations: <none> PolicyRule: Resources Non-Resource URLs Resource Names Verbs --------- ----------------- -------------- ----- pods [] [] [get delete list] statefulset [] [] [get delete list] statefulsets [] [] [get delete list] pods.apps [] [] [get delete list] statefulset.apps [] [] [get delete list] statefulsets.apps [] [] [get delete list]
oc describe -f role-binding.yaml
Name: pods-list Labels: <none> Annotations: <none> Role: Kind: Role Name: pods-list Subjects: Kind Name Namespace ---- ---- --------- ServiceAccount f5-dssm spk-utilities
Obtain the NAME of the current dSSM Helm release.
helm listIn this example, the dSSM Helm release NAME is f5-dssm:
NAME NAMESPACE REVISION UPDATED STATUS CHART f5-dssm spk-utilities 1 2021-10-13 08:33:11 deployed f5-dssm-0.9.0
Verify the
dssm_overrides.yamlcontents.cat dssm_overrides.yaml
Sample Output:
Upgrade the dSSM database Pods using the newer version Helm chart.
Note: The timeout value is a precaution; cluster resources may cause the process to go beyond the default 300 seconds.
helm upgrade f5-dssm <chart> -f dssm-values.yaml -n <namespace> --timeout 800s
In this example, the Helm chart version is 1.0.15-0.1.3:
helm upgrade f5-dssm tar/f5-dssm-1.0.15-0.1.5.tgz -f dssm_overrides.yaml -n spk-utilities --timeout 800s
To monitor the upgrade status, in the second shell, view the dssm-upgrade-hook container logs.
oc logs -f dssm-upgrade-hook
The upgrade logs should begin similar to the following:
HELM-HOOK IS RUNNING UPGRADING SENTINELS Namespace is spk-utilities dssm-upgrade-hook IS RUNNING
The upgrade logs should end similar to the following:
DONE UPGRADING Helm-hook pod is going down pod "dssm-upgrade-hook" deleted
Post-upgrade status¶
Use the steps below to ensure the dSSM software upgrade was successful.
List the REVISION ( version) of the dSSM Helm releases.
helm history f5-dssm
REVISION STATUS CHART APP VERSION DESCRIPTION 1 superseded f5-dssm-0.16.1 v0.16.1 Install complete 2 deployed f5-dssm-0.22.1 v0.22.1 Upgrade complete
Verify the dSSM Pod STATUS is currently Running.
oc get pods -n spk-utilities
NAME READY STATUS RESTARTS AGE f5-cert-manager-69bfff664d-pcg6b 1/1 Running 0 3h42m f5-cert-manager-cainjector-67f8d64694-nccl8 1/1 Running 0 3h42m f5-cert-manager-webhook-845c6bc784-f2l2c 1/1 Running 0 3h42m f5-crdconversion-6df5b56f49-x6ggc 2/2 Running 0 4h6m f5-dssm-db-0 3/3 Running 0 50m f5-dssm-db-1 3/3 Running 0 50m f5-dssm-db-2 3/3 Running 0 51m f5-dssm-sentinel-0 3/3 Running 0 49m f5-dssm-sentinel-1 3/3 Running 0 48m f5-dssm-sentinel-2 3/3 Running 0 48m f5-rabbit-64c8f87d68-cc9zv 2/2 Running 0 80m f5-spk-cwc-57584d85bd-hxs2r 3/3 Running 0 66m f5-toda-fluentd-f68f5d9cd-gjc52 1/1 Running 0 17h
Verify the f5-dssm-store version of the dSSM Pods.
oc describe pods | grep Image: | grep -i dssm
Image: artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.22.1 Image: artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.22.1 Image: artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.22.1 Image: artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.22.1 Image: artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.22.1 Image: artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.22.1
Verify the dSSM Pod STATUS is currently Running.
Note: It may take a few minutes for the rollback to complete.
oc get pods
NAME READY STATUS f5-dssm-db-0 2/2 Running f5-dssm-db-1 2/2 Running f5-dssm-db-2 2/2 Running f5-dssm-sentinel-0 2/2 Running f5-dssm-sentinel-1 2/2 Running f5-dssm-sentinel-2 2/2 Running
Log in to the dSSM database (DB).
oc exec -it f5-dssm-db-0 -- bash
Enter the Redis command line interface (CLI).
redis-cli --tls --cert /etc/ssl/certs/dssm-cert.crt \ --key /etc/ssl/certs/dssm-key.key \ --cacert /etc/ssl/certs/dssm-ca.crt
List the DB entries. These entries should be the same as the pre-upgrade check.
KEYS *1) "0073c3b6eft_dns4610.144.175.221" 2) "0073c3b6eft_dns4610.144.175.222" 3) "0073c3b6eft_dns4610.144.175.224" 4) "0073c3b6eft_dns4610.144.175.223" 5) "0073c3b6eft_dns4610.144.175.220"
Delete the
RoleandRoleBindingobjects.oc delete -f role-binding.yaml
oc delete -f role.yaml
Rollback¶
If the dSSM database is not performing as expected after the upgrade, rollback to the previous dSSM database version using the steps below:
List the current version of the dSSM database.
helm list -n spk-utilities
In this example, the dSSM database version is v.22.1 and the REVISION version is 2:
NAME NAMESPACE REVISION STATUS CHART APP VERSION f5-dssm spk-utilities 2 deployed f5-dssm-0.22.1 v0.22.1
To grant the dssm-upgrade-hook container access the K8S API, create two YAML files with the following code, and set the
namespaceparameter to the dSSM installation Project: Important: The dssm-upgrade-hook will fail to complete the upgrade without proper access to the K8S API.Example Role
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: pods-list namespace: spk-utilities rules: - apiGroups: ["", "apps"] resources: ["pods", "statefulsets", "statefulset"] verbs: ["get", "delete", "list", "watch"]
Example RoleBinding
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: pods-list subjects: - kind: ServiceAccount name: default namespace: spk-utilities roleRef: kind: Role name: pods-list apiGroup: rbac.authorization.k8s.io
Create the
RoleandRoleBindingobjects using example contents in the previous step.oc create -f role.yaml
oc create -f role-binding.yaml
Rollback the dSSM database to the previous REVISION (installation version).
In this example, the previous REVISION is 1:
helm rollback f5-dssm 1
List the Helm REVISION (installation versions) of the dSSM database.
helm history f5-dssm
REVISION STATUS CHART APP VERSION DESCRIPTION 1 superseded f5-dssm-0.16.1 v0.16.1 Install complete 2 superseded f5-dssm-0.22.1 v0.22.1 Upgrade complete 3 deployed f5-dssm-0.16.1 v0.16.1 Rollback to 1
Verify the dSSM Pod STATUS is currently Running.
Note: It may take a few minutes for the rollback to complete.
oc get pods
NAME READY STATUS f5-dssm-db-0 2/2 Running f5-dssm-db-1 2/2 Running f5-dssm-db-2 2/2 Running f5-dssm-sentinel-0 2/2 Running f5-dssm-sentinel-1 2/2 Running f5-dssm-sentinel-2 2/2 Running
Feedback
Provide feedback to improve this document by emailing spkdocs@f5.com.
Supplemental