#Upgrade Troubleshooting

Use this page when preflight reports a blocking check, CVO reports Ready=False, Reconciling=True, or Stalled=True, or a Core or Aligned module does not reach its target version.

#Inspect Upgrade State

Replace <cluster> with global or the workload cluster name:

kubectl -n cpaas-system get cvsh <cluster> \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

kubectl -n cpaas-system get cvsh <cluster> \
  -o jsonpath='{.status.preflight.observedAt}{"\n"}{range .status.preflight.checks[*]}{.name}{"\t"}{.policy}{"\t"}{.state}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

kubectl -n cpaas-system get cvsh <cluster> \
  -o jsonpath='{range .status.history[*]}{.version}{"\t"}{.state}{"\t"}{.startedTime}{"\t"}{.completionTime}{"\n"}{end}'

Resolve the first failing preflight check or the relevant Ready, Reconciling, or Stalled condition before changing the upgrade request.

#Handle Preflight Blocks

A ResourcePatch records a customization applied to a platform resource. The upgrade refuses to overwrite one unless it is exempted for the target version, so ResourcePatchUpgradeable is the most frequent block on a customized environment.

If ResourcePatchUpgradeable fails with reason=UnexemptResourcePatches, inspect the named ResourcePatch and add the required target-version exemption only after reviewing the patch:

kubectl -n cpaas-system get cvsh <cluster> \
  -o jsonpath='{range .status.preflight.checks[?(@.name=="ResourcePatchUpgradeable")]}{.state}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

kubectl get resourcepatches <rp-name> -o yaml
kubectl annotate resourcepatches <rp-name> \
  config.cpaas.io/exempt-for-ver=<target-version> \
  --overwrite

The exemption annotation key is config.cpaas.io/exempt-for-ver, and its value is the target version. The exemption is matched on the major and minor version, so an exemption recorded for one patch version also covers the other patch versions of the same minor release.

Exempting a patch tells the upgrade to leave that customization in place. It does not verify that the customization is still valid on the target version — confirm that with the technical support contact who supplied it.

#Resolve Other Common Preflight Blocks

List every check that is not Passed with the command in Inspect Upgrade State, then resolve each one:

Do not force an upgrade past a failed check. VersionUpgradePath, KubernetesVersionSupported, and AdminAckRequired gate the supported upgrade path itself, and an upgrade that bypasses them reaches a state the platform has not validated. If a block cannot be resolved from this page, contact technical support before continuing.

#Handle Administrator Acknowledgement Gates

If AdminAckRequired fails, inspect the keys supplied by the target release:

kubectl -n cpaas-system get configmap admin-gates -o yaml

Complete the action described by the applicable gate. For a Kubernetes 1.35 or later node-readiness gate, complete Kubernetes 1.35 or Later Node Readiness on every production node.

The first gate introduced for that requirement, for the ACP 4.4 and Kubernetes 1.35 target, is ack-4.4-kubernetes-1.35-kernel-update. Treat it as an example — a later target release can supply a different key, so always read the key out of admin-gates rather than reusing this one.

Copy the applicable key from admin-gates, record the acknowledgement in admin-acks, and rerun preflight:

ACK_KEY='<key-from-admin-gates>'
kubectl -n cpaas-system patch configmap admin-acks --type merge \
  -p "{\"data\":{\"${ACK_KEY}\":\"true\"}}"

bash upgrade.sh --preflight

Confirm that the check now passes:

kubectl -n cpaas-system get cvsh <cluster> \
  -o jsonpath='{range .status.preflight.checks[?(@.name=="AdminAckRequired")]}{.state}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

#Recover a Missing Aligned Package

An unavailable Aligned package does not change the application's current installed state. Depending on the package type and reconciliation stage, CVO can report the problem through Ready=False or Reconciling=True rather than Stalled=True.

#Aligned Cluster Plugins

If a ClusterVersionShadow condition contains required ready ModulePluginConfig for installed platform-aligned component, identify the target ProductManifest and the named ModulePluginConfig:

kubectl get productmanifest v<target-version> -o yaml

kubectl get modulepluginconfig <modulepluginconfig-name> \
  -o jsonpath='{.spec.image}{"\n"}{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

For a ModulePlugin channel, artifactStatus: Absent in the target ProductManifest means that the matching target ModulePluginConfig is not fully Ready. It does not mean that the currently installed application became absent, and it does not by itself prove that the image is missing from the registry.

Use the ModulePluginConfig condition and the exact image in .spec.image to select the recovery:

• For an application in ACP Upgrade to v4.5, copy its package into the target Core Package's plugins/ directory. With the platform built-in Registry, rerun upgrade.sh --only-sync-image. With an external registry, rerun both modes in Prepare the Target-Version Payload.
• For another Aligned cluster plugin, publish the target package to the global tier with violet push.

#Aligned Operators

Aligned operators do not use ModulePluginConfig. If an installed operator does not advance, inspect its Subscription, target InstallPlan, catalog, and ModuleInfo on the affected cluster:

kubectl get subscription -A
kubectl get installplan -A
kubectl get moduleinfo -l cpaas.io/cluster-name=<cluster>

If the target operator package is missing, publish it to every cluster where that operator is installed. If only one workload cluster is missing it, push to that cluster with --clusters "<workload-cluster-name>". If the package exists, resolve the Subscription, InstallPlan, catalog, or ModuleInfo condition actually reported by CVO.

Continue observing the existing request after correcting the problem. CVO retries automatically; do not clear or resubmit desiredUpdate.

#Diagnose a Module That Does Not Advance

kubectl get moduleinfo -l cpaas.io/cluster-name=<cluster> \
  -o custom-columns='MODULE:.metadata.labels.cpaas\.io/module-name,CURRENT:.status.version,TARGET:.spec.version,NEW:.status.availableVersions[0].version,PHASE:.status.phase'

kubectl get moduleinfo <name> \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

Inspect the named resource and related Events. For image-pull failures, verify registry reachability, CA trust, pull credentials, and the referenced manifest from the affected node.

#Collect Evidence Before Escalation

Capture the cvsh conditions, preflight checks, stages and history, the affected ModuleInfo or ModulePluginConfig, related Events, and the exact source and target versions. Do not include platform tokens, registry passwords, or Secret contents.