SUC Upgrade Troubleshooting

Note

This is not official documentation for AutomationSuite

Issue1

The upgrade timeout with Error: cannot execute upgrade plan on agent nodes: context canceled, however, all the nodes were upgraded. 

Workaround

The Rerun upgrade passed without any issues.

Symptoms

Upgrade log: From the first node upgrade log, all the nodes upgraded but failed with context canceled for the agent plan.

[INFO] [2023-10-11T12:51:12+0000]: Running upgrade on all nodes...
Error: cannot execute upgrade plan on agent nodes: context canceled
[INFO] [2023-10-11T13:51:12+0000]: Node details:
[INFO] [2023-10-11T13:51:12+0000]: NAME      STATUS   ROLES                       AGE     VERSION
agent0    Ready    <none>                      6h44m   v1.26.5+rke2r1
server0   Ready    control-plane,etcd,master   7h18m   v1.26.5+rke2r1
server1   Ready    control-plane,etcd,master   7h2m    v1.26.5+rke2r1
server2   Ready    control-plane,etcd,master   6h53m   v1.26.5+rke2r1
[INFO] [2023-10-11T13:51:12+0000]: Refer the log files in /opt/UiPathAutomationSuite/_upgrade/UiPath_Installer/Modules/../upgrade-logs/upgrade-2023.10.0-rc.12
[INFO] [2023-10-11T13:51:16+0000]: Rke upgrade information is available at system-upgrade namespace. Use kubectl command to get the logs and events.
[INFO] [2023-10-11T13:51:16+0000]: Logs also present in the corresponding node at /opt/UiPathAutomationSuite/_autoupgrade
^[[0;31m[ERROR][2023-10-11T13:51:16+0000]:^[[0m Upgrade failed. Please fix the errors and try again.

From the agent node log (upgraded succeeded)

/opt/UiPathAutomationSuite/<version>/installer/logfile-apply-a-upgrade-<version>-on-agent0-with-<some-random-hash>.log

[INFO] [2023-10-12T12:42:53+0000]: Infra installed successfully
[INFO] [2023-10-12T12:42:53+0000]: Validate rke2 images update in the upgrade hook
[INFO] [2023-10-12T12:42:53+0000]: checking all rke2 pods are up and running in the current node
pod/kube-proxy-agent0 condition met
[INFO] [2023-10-12T12:42:54+0000]: Uncordoning node in the upgrade hook
Checking if node  is ready to run kubectl command.
Node is ready to accept kubectl command
Enable IP Forwarding...
Either file /etc/sysctl.conf not present or file is not writable. Enabling ip forward using /proc/sys/net/ipv4/ip_forward...
Uncordon agent0 ...
label "nodejanitor/skip" not found.
node/agent0 not labeled
node/agent0 already uncordoned
node/agent0 annotated
[INFO] [2023-10-12T12:42:54+0000]: Upgrade successfully completed.