Upgrading dSSM

Overview

The Service Proxy for Kubernetes (SPK) distributed Session State Management (dSSM) Sentinel and DB Pods can be upgraded using the typical Helm upgrade process. However, to ensure the process completes without service interruption, a custom dssm-upgrade-hook container is deployed during the upgrade, and requires additional permissions to complete the upgrade tasks. The upgrade process maintains all of the dSSM DB Pod session state data.

_images/spk_info.png Note: If preserving data is not required, refer to the Quick Upgrade section to properly uninstall and upgrade the dSSM installation.

This document guides you through upgrading the dSSM database, and verifying the results.

Requirements

Ensure you have:

  • A running SPK dSSM Database installation.
  • Uploaded the f5-dssm-upgrader image with the SPK Software installation.
  • A newer version of the SPK dSSM Helm chart.
  • A workstation with Helm installed.

Procedures

Use the procedures below to upgrade the dSSM database, verify the results, and if required, rollback to the previous installation version.

File permissions

Beginning in SPK version 1.4.0, the dSSM containers run as non-root user ID 7053. To access the previously created PVC data on the underlying storage system, the mapped PVC files must have the user ID (UID) and group ID (GID) changed to the 7053. Use the steps below to obtain and modify the PVC file UID/GIDs on the storage system.

_images/spk_warn.png Important: The dSSM upgrade will fail if the PVC file UID/GIDs are not modified.

  1. Switch to the dSSM Pod Project:

    oc project <name>
    

    In this example, the dSSM Pods are in the spk-utilities Project:

    oc project spk-utilities
    
  2. Obtain the names of the dSSM PVCs:

    oc get pvc | awk '{print $1, " ", $3}'
    
    VOLUME
    data-f5-dssm-db-0   pvc-5a591864-86b7-4733-9812-ac05a9723685
    data-f5-dssm-db-1   pvc-9b69417d-5b43-4a5b-9b15-b7dc185157cd
    data-f5-dssm-db-2   pvc-6a5fd7a8-7dac-46ed-bf12-aa0c7f1ff13a
    
  3. Obtain the Server and the mapped PVC file Path:

    oc describe pv <pvc> | grep -iE 'server|path'
    

    In this example, the first PVC pvc-5a591864-86b7-4733-9812-ac05a9723685 is described:

    oc describe pv pvc-5a591864-86b7-4733-9812-ac05a9723685 | grep -iE 'server|path'
    
  4. The command output shows the PVC maps to a server named provisioner.ocp.f5.com, and is in the /home/kni/nfs_share/ocp directory:

    Server:    provisioner.ocp.f5.com
    Path:      /home/kni/nfs_share/ocp/spk-utilities-data-f5-dssm-db-0-pvc-5a591864-86b7-4733-9812-ac05a9723685
    

    The complete list of mapped dSSM PVCs will appear similar to the following:

    /home/kni/nfs_share/ocp/spk-utilities-data-f5-dssm-db-0-pvc-5a591864-86b7-4733-9812-ac05a9723685
    /home/kni/nfs_share/ocp/spk-utilities-data-f5-dssm-db-1-pvc-9b69417d-5b43-4a5b-9b15-b7dc185157cd
    /home/kni/nfs_share/ocp/spk-utilities-data-f5-dssm-db-2-pvc-6a5fd7a8-7dac-46ed-bf12-aa0c7f1ff13a
    
  5. Use Secure Shell (SSH) to access the storage server:

    ssh root@<server>
    

    In this example, the server hostname is provisioner.ocp.f5.com:

    ssh root@provisioner.ocp.f5.com
    
  6. Modify the mapped PVC file using the new UID/GID:

    sudo chown -R 7053:7053 /path/to/file/*
    

    The complete list of modified files will appear similar to the following:

    sudo chown -R 7053:7053  /home/kni/nfs_share/ocp/spk-utilities-data-f5-dssm-db-0-pvc-5a591864-86b7-4733-9812-ac05a9723685/*
    sudo chown -R 7053:7053  /home/kni/nfs_share/ocp/spk-utilities-data-f5-dssm-db-0-pvc-9b69417d-5b43-4a5b-9b15-b7dc185157cd/*
    sudo chown -R 7053:7053  /home/kni/nfs_share/ocp/spk-utilities-data-f5-dssm-db-0-pvc-6a5fd7a8-7dac-46ed-bf12-aa0c7f1ff13a/*
    
  7. Verify the new UID/GIDs:

    ls -arlt /path/to/file
    

    The file permissions should appear as follows:

    ls -arlt /home/kni/nfs_share/ocp/spk-utilities-data-f5-dssm-db-0-pvc-5a591864-86b7-4733-9812-ac05a9723685/*
    
    drwxrwxrwx.   2 nobody nobody    62 Oct 11 15:00 .
    drwxrwxr-x. 510 nobody nobody 49152 Oct 11 15:01 ..
    -rw-r--r--.   1   7053   7053  6554 Oct 11 15:12 appendonly.aof
    -rw-r--r--.   1   7053   7053   175 Oct 11 15:00 dump.rdb
    -rw-r--r--.   1   7053   7053   477 Oct 11 15:00 redis.conf
    

Pre-upgrade status

Use the step below to verify the dSSM Pod cluster status, software version and persisted data. This will be useful to ensure the upgrade is succesful.

  1. Ensure the dSSM installation Project is selected:

    oc project <name>
    

    In this example, the dSSM Pods are in the spk-utilities Project:

    oc project spk-utilities
    
  2. Verify the STATUS of the dSSM Pods is Running:

    oc get pods
    
    NAME                              READY   STATUS    RESTARTS  
    f5-dssm-db-0                      2/2     Running   0       
    f5-dssm-db-1                      2/2     Running   0     
    f5-dssm-db-2                      2/2     Running   0   
    f5-dssm-sentinel-0                2/2     Running   0        
    f5-dssm-sentinel-1                2/2     Running   0       
    f5-dssm-sentinel-2                2/2     Running   0      
    
  3. Verify the f5-dssm-store version:

    oc describe pods | grep Image: | grep -i dssm
    

    In this example, the f5-dssm-store is v1.6.1:

     Image:         artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.6.1
     Image:         artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.6.1
     Image:         artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.6.1
     Image:         artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.6.1
     Image:         artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.6.1
     Image:         artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.6.1
    
  4. Log in to the dSSM database (DB):

    oc exec -it f5-dssm-db-0 -- bash
    
  5. Enter the Redis command line interface (CLI):

    redis-cli --tls --cert /etc/ssl/certs/dssm-cert.crt \
    --key /etc/ssl/certs/dssm-key.key \
    --cacert /etc/ssl/certs/dssm-ca.crt
    
  6. List the DB entries. The entries should be present after the upgrade.

    KEYS *
    
    1) "0073c3b6eft_dns4610.144.175.221"
    2) "0073c3b6eft_dns4610.144.175.222"
    3) "0073c3b6eft_dns4610.144.175.224"
    4) "0073c3b6eft_dns4610.144.175.223"
    5) "0073c3b6eft_dns4610.144.175.220"
    

Software upgrade

Use the steps below to upgrade the dSSM Sentinel and DB Pods.

Note: The dssm-upgrade-hook container logs valuable diagnostic data, opening a second shell to view the data is recommended.

  1. Ensure the dSSM installation Project is selected:

    oc project <name>
    

    In this example, the dSSM Pods are in the spk-utilities Project:

    oc project spk-utilities
    
  2. The f5-dssm-upgrader image is provided with the SPK Software, and must be referenced for the upgrade using the dssm-values.yaml file below:

    Note: Replace the local.registry.com value below with the domain name of your local image registry.

    dssmUpgrader:
      image:
        repository: "local.registry.com"
    
  3. To grant the dssm-upgrade-hook container access the K8S API, create two YAML files with the following code, and set the namespace parameter to the dSSM installation Project:

    _images/spk_warn.png Important: The dssm-upgrade-hook will fail to complete the upgrade without proper access to the K8S API.

    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: pods-list
      namespace: spk-utilities
    rules:
    - apiGroups: [""]
      resources: ["pods"]
      verbs: ["get", "delete"]
    
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: pods-list
    subjects:
    - kind: ServiceAccount
      name: default
      namespace: spk-utilities
    roleRef:
      kind: Role
      name: pods-list
      apiGroup: rbac.authorization.k8s.io
    
  4. Create the Role and RoleBinding objects:

    oc create -f role.yaml
    
    oc create -f role-binding.yaml
    
  5. Verify the Role and RoleBinding objects have been created:

    oc describe -f role.yaml
    
    Name:         pods-list
    Labels:       <none>
    Annotations:  <none>
    PolicyRule:
      Resources  Non-Resource URLs  Resource Names  Verbs
      ---------  -----------------  --------------  -----
      pods       []                 []              [get delete]
    
    oc describe -f role-bind.yaml
    
    Name:         pods-list
    Labels:       <none>
    Annotations:  <none>
    Role:
      Kind:  Role
      Name:  pods-list
    Subjects:
      Kind            Name     Namespace
      ----            ----     ---------
      ServiceAccount  default  spk-utilities
    
  6. Obtain the NAME of the current dSSM Helm release:

    helm list
    

    In this example, the dSSM Helm release NAME is f5-dssm:

    NAME       NAMESPACE        REVISION   UPDATED               STATUS      CHART
    f5-dssm    spk-utilities   1          2021-10-13 08:33:11   deployed    f5-dssm-0.9.0
    
  7. Upgrade the dSSM database Pods using the newer version Helm chart:

    Note: The timeout value is a precaution; cluster resources may cause the process to go beyond the default 300 seconds.

    helm upgrade f5-dssm <chart> -f dssm-values.yaml --timeout 800s
    

    In this example, the Helm chart version is 0.22.1:

    helm upgrade f5-dssm f5-dssm-0.22.1.tgz -f dssm-values.yaml --timeout 800s
    
  8. To monitor the upgrade status, in the second shell, view the dssm-upgrade-hook container logs:

    oc logs -f dssm-upgrade-hook
    

    The upgrade logs should begin similar to the following:

    HELM-HOOK IS RUNNING
    UPGRADING SENTINELS
    Namespace is spk-utilities
    dssm-upgrade-hook IS RUNNING
    

    The upgrade logs should end similar to the following:

    DONE UPGRADING
    Helm-hook pod is going down
    pod "dssm-upgrade-hook" deleted
    

Post-upgrade status

Use the steps below to ensure the dSSM software upgrade was successful.

  1. List the REVISION ( version) of the dSSM Helm releases:

    helm history f5-dssm
    
    REVISION    STATUS         CHART             APP VERSION     DESCRIPTION
    1           superseded     f5-dssm-0.16.1    v0.16.1         Install complete
    2           deployed       f5-dssm-0.22.1    v0.22.1         Upgrade complete
    
  2. Verify the dSSM Pod STATUS is currently Running:

    oc get pods
    
    NAME                      READY   STATUS 
    f5-dssm-db-0              2/2     Running
    f5-dssm-db-1              2/2     Running  
    f5-dssm-db-2              2/2     Running 
    f5-dssm-sentinel-0        2/2     Running 
    f5-dssm-sentinel-1        2/2     Running
    f5-dssm-sentinel-2        2/2     Running
    
  3. Verify the f5-dssm-store version of the dSSM Pods:

    oc describe pods | grep Image: | grep -i dssm
    
     Image:         artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.22.1
     Image:         artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.22.1
     Image:         artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.22.1
     Image:         artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.22.1
     Image:         artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.22.1
     Image:         artifactory.f5net.com/f5-mbip-docker/f5-dssm-store:v1.22.1
    
  4. Verify the dSSM Pod STATUS is currently Running:

    Note: It may take a few minutes for the rollback to complete.

    oc get pods
    
    NAME                      READY   STATUS 
    f5-dssm-db-0              2/2     Running
    f5-dssm-db-1              2/2     Running  
    f5-dssm-db-2              2/2     Running 
    f5-dssm-sentinel-0        2/2     Running 
    f5-dssm-sentinel-1        2/2     Running
    f5-dssm-sentinel-2        2/2     Running
    
  5. Log in to the dSSM database (DB):

    oc exec -it f5-dssm-db-0 -- bash
    
  6. Enter the Redis command line interface (CLI):

    redis-cli --tls --cert /etc/ssl/certs/dssm-cert.crt \
    --key /etc/ssl/certs/dssm-key.key \
    --cacert /etc/ssl/certs/dssm-ca.crt
    
  7. List the DB entries. These entries should be the same as the pre-upgrade check.

    KEYS *
    
    1) "0073c3b6eft_dns4610.144.175.221"
    2) "0073c3b6eft_dns4610.144.175.222"
    3) "0073c3b6eft_dns4610.144.175.224"
    4) "0073c3b6eft_dns4610.144.175.223"
    5) "0073c3b6eft_dns4610.144.175.220"
    
  8. Delete the Role and RoleBinding objects:

    oc delete -f role-binding.yaml
    
    oc delete -f role.yaml
    

Rollback

If the dSSM database is not performing as expected after the upgrade, rollback to the previous dSSM database version using the steps below:

  1. List the current version of the dSSM database:

    helm list
    

    In this example, the dSSM database version is v.22.1 and the REVISION version is 2:

    NAME         NAMESPACE      	REVISION    STATUS      CHART                   APP VERSION
    f5-dssm      spk-utilities   2           deployed    f5-dssm-0.22.1          v0.22.1
    
  2. Rollback the dSSM database to the previous REVISION (installation version):

    helm rollback f5-dssm <revision>
    

    In this example, the previous REVISION is 1:

    helm rollback f5-dssm 1
    
  3. List the Helm REVISION (installation versions) of the dSSM database:

    helm history f5-dssm
    
    REVISION    STATUS         CHART             APP VERSION     DESCRIPTION
    1           superseded     f5-dssm-0.16.1    v0.16.1         Install complete
    2           superseded     f5-dssm-0.22.1    v0.22.1         Upgrade complete
    3           deployed       f5-dssm-0.16.1    v0.16.1         Rollback to 1
    
  4. Verify the dSSM Pod STATUS is currently Running:

    Note: It may take a few minutes for the rollback to complete.

    oc get pods
    
    NAME                      READY   STATUS 
    f5-dssm-db-0              2/2     Running
    f5-dssm-db-1              2/2     Running  
    f5-dssm-db-2              2/2     Running 
    f5-dssm-sentinel-0        2/2     Running 
    f5-dssm-sentinel-1        2/2     Running
    f5-dssm-sentinel-2        2/2     Running
    

Quick Upgrade

The quick upgrade section provides a much easier way to upgrade the dSSM database if preserving data is not a requirement. Use the steps below to properly uninstall the current dSSM Database installation and then reinstall using Helm.

  1. List the dSSM Helm release:

    helm list -n <project>
    

    In this exammple, the dSSM database release f5-dssm is installed in the spk-utils Project:

    helm list -n <project>
    
    NAME     NAMESPACE    REVISION   STATUS     CHART           APP VERSION
    f5-dssm  spk-utils    1          deployed   f5-dssm-0.16.1  v0.16.1
    
  2. Uninstall the dSSM installation:

    helm uninstall f5-dssm -n spk-utils
    

    The commaznd output will appear similar to the following:

    release "f5-dssm" uninstalled
    
  3. List the dSSM PVCs:

    oc get pvc -n spk-utils
    
    NAME                       STATUS   VOLUME                                   
    data-f5-dssm-db-0          Bound    pvc-933c17ae-4378-4eac-8d09-65848a1e164e
    data-f5-dssm-db-1          Bound    pvc-c843c33b-c277-46f2-bcb1-4ee5db76ea4b
    data-f5-dssm-db-2          Bound    pvc-d0f5441b-0e0c-4385-b558-a84e15fc44a9
    
  4. Delete each of the PVCs using the PVC NAME:

    oc delete pvc data-f5-dssm-db-0 -n spk-utils
    

    The command output will appear similar to the following:

    persistentvolumeclaim "data-f5-dssm-db-0" deleted
    
  5. Once all of the PVCs have been deleted, reinstall the dSSM DBs using the dSSM Database installation guide.

Feedback

Provide feedback to improve this document by emailing spkdocs@f5.com.

Supplemental