Skip to main content

Maintenance and Upgrades

This page explains how to maintain an existing AI Cockpit: Smart Engineering on-premise deployment after the initial installation.

The most common operational tasks are:

  • applying a new chart version
  • updating the application image version
  • rotating certificates and ingress settings
  • validating storage, database, and queue health
  • preparing backups before changes
  • rolling back when an upgrade fails

Before applying changes​

Before you update the deployment, confirm:

  • the cluster is healthy
  • PostgreSQL and Valkey are running normally
  • the current release name and namespace are known
  • the current values.yaml file is available
  • you have a recent backup of the database

Useful commands:

kubectl get pods -n aic-modernization
kubectl get jobs -n aic-modernization
helm list -n aic-modernization
helm get values aic-modernization -n aic-modernization

Keep the values file under change control​

Do not rely only on the original install command from AWS Marketplace.

Keep the effective deployment values in version control so you can:

  • review what changed between releases
  • reproduce the environment
  • compare current and target configuration
  • roll back more safely

If needed, export the current values before an upgrade:

helm get values aic-modernization -n aic-modernization -o yaml > current-values.yaml

Upgrading the deployment​

When a new chart or product version is available, update the deployment with helm upgrade.

Example:

helm upgrade aic-modernization \
  oci://709825985650.dkr.ecr.us-east-1.amazonaws.com/compass-uol/ai-cockpit/smart-engineering-chart/aic-modernization \
  --version <new-chart-version> \
  --namespace aic-modernization \
  -f values.yaml

For upgrades, the most important decision is how Helm should handle values from the current release.

Reuse the current values​

Use --reuse-values when you want to keep the values already stored in the current release and only apply the explicit changes passed in the upgrade command.

Example:

helm upgrade aic-modernization \
  oci://709825985650.dkr.ecr.us-east-1.amazonaws.com/compass-uol/ai-cockpit/smart-engineering-chart/aic-modernization \
  --version <new-chart-version> \
  --namespace aic-modernization \
  --reuse-values

This is useful when:

  • the release already has the correct runtime configuration
  • you only want to change the chart version
  • you want to minimize accidental value drift during an upgrade

Reset to chart defaults​

Use --reset-values when you want Helm to ignore the stored release values and start again from the target chart defaults plus the values you pass in the command.

Example:

helm upgrade aic-modernization \
  oci://709825985650.dkr.ecr.us-east-1.amazonaws.com/compass-uol/ai-cockpit/smart-engineering-chart/aic-modernization \
  --version <new-chart-version> \
  --namespace aic-modernization \
  --reset-values \
  -f values.yaml

This is useful when:

  • you want the release to match a reviewed values.yaml exactly
  • older stored values may no longer be valid
  • you are standardizing the environment after several manual changes

Which approach to choose​

As a rule:

  • use --reuse-values for small and low-risk upgrades
  • use --reset-values together with a reviewed values.yaml for controlled configuration changes or major version transitions

What happens during an upgrade​

During helm upgrade, the chart may update:

  • application deployments
  • image tags
  • runtime configuration
  • ingress configuration
  • gateway configuration
  • PostgreSQL or Valkey settings

The chart also runs a migration job after install and upgrade.

After the upgrade, verify:

kubectl get pods -n aic-modernization
kubectl get jobs -n aic-modernization
kubectl logs job/aic-modernization-migrate -n aic-modernization

Validate the application after an upgrade​

After applying an update, validate:

  • API health
  • gateway health, if enabled
  • frontend access
  • S3 upload flow
  • Bedrock access
  • background task execution

Recommended checks:

kubectl port-forward svc/aic-modernization-api -n aic-modernization 3000:3000
curl http://localhost:3000/health
curl http://localhost:3000/config/runtime

If the gateway is enabled:

kubectl port-forward svc/aic-modernization-gateway -n aic-modernization 8080:8080
curl http://localhost:8080/health

Backups and data protection​

Before changing chart versions, image versions, database settings, or storage settings, take a backup of the PostgreSQL database.

At minimum, maintain:

  • regular PostgreSQL backups
  • persistent volume snapshot procedures, if your storage platform supports them
  • a copy of the current values.yaml

If you are using external PostgreSQL or external Valkey services, follow the backup policy for those managed services.

warning

Do not treat Helm alone as a backup strategy. Helm stores release state, but it does not replace database backups.

Rolling back​

If an upgrade fails and the new version should not remain in production, check the release history:

helm history aic-modernization -n aic-modernization

Then roll back to the previous known-good revision:

helm rollback aic-modernization <revision> -n aic-modernization

After rollback, validate the same health and runtime endpoints again.

If the failure included a partial database migration, review the migration logs before retrying another upgrade.

Common maintenance tasks​

Rotating ingress certificates​

If the TLS certificate changes:

  • update the Kubernetes TLS secret used by ingress
  • confirm the ingress still references the correct secret name
  • validate the frontend, API, and gateway hosts after the change

Updating allowed origins​

If the browser origin changes, update:

  • runtime.commonEnv.ALLOWED_ORIGINS
  • gateway.env.CORS_ALLOW_ORIGINS, if the gateway is exposed directly

Apply the change with helm upgrade.

Changing S3 encryption settings​

If the target S3 bucket policy changes:

  • keep S3_UPLOAD_SERVER_SIDE_ENCRYPTION=AES256 for the default S3-managed encryption path
  • switch to aws:kms only if the bucket requires KMS and the runtime IAM role can use the KMS key

Updating Bedrock models​

If you change models, update:

  • bedrock.modelId
  • bedrock.modelIdSmall

Then validate that:

  • the target model IDs are available in the buyer account
  • the runtime IAM role can invoke those models

Operational troubleshooting checklist​

If the system is unhealthy after an update, check:

  • pod restart loops
  • failed migration jobs
  • ingress misconfiguration
  • service account or IRSA changes
  • S3 permission changes
  • Bedrock model access changes
  • database connection secret changes
  • Valkey connectivity issues

Useful commands:

kubectl describe pod <pod-name> -n aic-modernization
kubectl logs deployment/aic-modernization-api -n aic-modernization --tail=200
kubectl logs deployment/aic-modernization-worker -n aic-modernization --tail=200
kubectl logs deployment/aic-modernization-gateway -n aic-modernization --tail=200

A practical maintenance routine is:

  1. Export the current Helm values
  2. Confirm backups are current
  3. Review the target chart version and configuration changes
  4. Run helm upgrade
  5. Check the migration job
  6. Validate /health and /config/runtime
  7. Run a small end-to-end documentation task

This keeps upgrades predictable and makes rollback simpler when something changes unexpectedly.