Secret Management

This topic provides instructions for creating and managing Kubernetes secrets for the Obliq SRE Agent deployment. The Helm chart allows you to either let it manage secrets automatically or provide your own pre-existing Kubernetes secrets.

Secret Types

Secret Type	Purpose	Chart Creates	Pre-existing Support
Global Secret	All environment variables	✅	✅
Image Pull Secret	Container registry access	✅	✅
Kubeconfig Secret	Kubernetes cluster access	✅	✅

Required Environment Variables

Core Variables

info

The core variables are required.

Variable	Description
OPENAI_API_KEY	OpenAI API key for AI services
KUBECONFIG	Kubernetes configuration for cluster access

AWS Integration

info

The AWS integration variables are required only with AWS integration. Otherwise, they are optional.

Variable	Description
AWS_ACCESS_KEY_ID	AWS access key for cloud services
AWS_SECRET_ACCESS_KEY	AWS secret access key
AWS_REGION	AWS region (default: us-east-1)
AWS_ROLE_ARN_AWS_MCP	IAM role for AWS MCP service
AWS_ROLE_ARN_EC2_CLOUDWATCH_ALARMS	IAM role for CloudWatch alarms

External Integrations

info

The external integration variables are required only with external integration. Otherwise, they are optional.

Variable	Description
SLACK_BOT_TOKEN	Slack bot token for notifications (xoxb-...)
SLACK_WEBHOOK_URL	Slack webhook URL for alerts
DD_API_KEY	DataDog API key for service graph engine
DD_APP_KEY	DataDog application key
JIRA_EMAIL	Jira user email for incident management
JIRA_API_TOKEN	Jira API token
JIRA_BASE_URL	Jira instance URL (default: https://avesha.atlassian.net)
PROMETHEUS_URL	Prometheus server URL (default: http://prometheus:9090)
PROMETHEUS_USER	Prometheus username (if auth enabled)
PROMETHEUS_PASSWORD	Prometheus password (if auth enabled)
LOKI_URL	Loki server URL (default: http://loki:3100)
LOKI_USERNAME	Loki username (if auth enabled)
LOKI_PASSWORD	Loki password (if auth enabled)
LOKI_TOKEN	Loki authentication token

Method 1: Using Pre-existing Global Secret

We recommend you this method. Create your own Kubernetes secret with all required environment variables.

Create the Secret

Minimal Deployment (Core Services Only)

kubectl create secret generic obliq-secrets \
  --namespace=avesha \
  --from-literal=OPENAI_API_KEY="sk-your-openai-key"

AWS Integration

kubectl create secret generic obliq-secrets \
  --namespace=avesha \
  --from-literal=OPENAI_API_KEY="sk-your-openai-key" \
  --from-literal=AWS_ACCESS_KEY_ID="your-aws-access-key" \
  --from-literal=AWS_SECRET_ACCESS_KEY="your-aws-secret-key" \
  --from-literal=AWS_REGION="us-east-1" \
  --from-literal=AWS_ROLE_ARN_AWS_MCP="arn:aws:iam::account:role/aws-mcp-role"

Full Integration (All Services)

kubectl create secret generic obliq-secrets \
  --namespace=avesha \
  --from-literal=OPENAI_API_KEY="sk-your-openai-key" \
  --from-literal=AWS_ACCESS_KEY_ID="your-aws-access-key" \
  --from-literal=AWS_SECRET_ACCESS_KEY="your-aws-secret-key" \
  --from-literal=AWS_REGION="us-east-1" \
  --from-literal=SLACK_BOT_TOKEN="xoxb-your-slack-token" \
  --from-literal=SLACK_WEBHOOK_URL="https://hooks.slack.com/services/your-webhook" \
  --from-literal=DD_API_KEY="your-datadog-api-key" \
  --from-literal=DD_APP_KEY="your-datadog-app-key" \
  --from-literal=JIRA_EMAIL="user@company.com" \
  --from-literal=JIRA_API_TOKEN="your-jira-api-token" \
  --from-literal=JIRA_BASE_URL="https://company.atlassian.net" \
  --from-literal=PROMETHEUS_URL="http://prometheus:9090" \
  --from-literal=PROMETHEUS_USER="admin" \
  --from-literal=PROMETHEUS_PASSWORD="your-prometheus-password" \
  --from-literal=LOKI_URL="http://loki:3100" \
  --from-literal=LOKI_USERNAME="admin" \
  --from-literal=LOKI_PASSWORD="your-loki-password"

Configure Chart to Use Existing Secret

Create a values file or use command line options as shown in the YAML file below.

# custom-values.yaml
global:
  globalSecret:
    create:
      enabled: false  # Don't create a new secret
    existing:
      enabled: true   # Use existing secret
      name: "obliq-secrets"  # Your secret name

Install with Pre-existing Secret

helm install obliq-sre-agent obliq-charts/obliq-sre-agent \
  --namespace avesha \
  --create-namespace \
  --set-file global.kubeconfig.content=./kubeconfig \
  --set global.globalSecret.existing.enabled=true \
  --set global.globalSecret.existing.name=obliq-secrets \
  --set global.globalSecret.create.enabled=false

Method 2: Using Pre-existing Image Pull Secret

If you already have a Docker registry secret, then follow the steps below.

Create Registry Secret

Create a registry secret if it is required.

kubectl create secret docker-registry registry-secret \
  --docker-server=avesha.azurecr.io \
  --docker-username=your-username \
  --docker-password=your-password \
  --docker-email=your-email \
  --namespace=avesha

Configure Chart

# custom-values.yaml
global:
  imagePullSecrets:
    - name: registry-secret
  imagePullSecretConfig:
    create:
      enabled: false  # Don't create new registry secret
    existing:
      enabled: true   # Use existing registry secret
      name: "registry-secret"

Service-Specific Environment Variables

Each service requires specific environment variables from the global secret.

Core Services

info

The core services are always enabled.

Service	Required Variables
backend	OPENAI_API_KEY, PORT, INFRA_AGENT_HOST, INFRA_AGENT_PORT
orchestrator	OPENAI_API_KEY, MCP_SERVERS, PORT
rca-agent	OPENAI_API_KEY, MCP_SERVERS, PORT
anomaly-detection	OPENAI_API_KEY, MCP_SERVERS, PORT
auto-remediation	OPENAI_API_KEY, MCP_SERVERS, PORT
incident-manager	OPENAI_API_KEY, MCP_SERVERS, PORT

Optional Services

info

Enable the optional services as required.

Service	Required Variables
aws-mcp	AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, AWS_ROLE_ARN_AWS_MCP
cloudwatch-mcp	AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION
prometheus-mcp	PROMETHEUS_URL, PROMETHEUS_USER, PROMETHEUS_PASSWORD
loki-mcp	LOKI_URL, LOKI_USERNAME, LOKI_PASSWORD, LOKI_TOKEN
slack-ingester	SLACK_BOT_TOKEN
service-graph-engine	DD_API_KEY, DD_APP_KEY, DD_SITE

Configuration Examples

Enable AWS Services

# Enable AWS-related services
aws-mcp:
  enabled: true
cloudwatch-mcp:
  enabled: true
aws-ec2-cloudwatch-alarms:
  enabled: true

Enable Observability Services

# Enable observability integrations
prometheus-mcp:
  enabled: true
loki-mcp:
  enabled: true

Enable External Integrations

# Enable external service integrations
slack-ingester:
  enabled: true
service-graph-engine:
  enabled: true

Verification and Troubleshooting

Check Secret Creation

• Verify the secret that you created using the following command:

  kubectl get secrets -n avesha

• Check secret content (keys only) using the following command:

  kubectl describe secret obliq-secrets -n avesha

• View secret data (base64 encoded) using the following command:

  kubectl get secret obliq-secrets -n avesha -o yaml

Verify Environment Variables in Pods

• Check the backend pod environment using the following command:

  kubectl exec -n avesha deployment/backend -- env | grep -E "(OPENAI|AWS|SLACK)" | sort

• Check specific service using the following command:

  kubectl exec -n avesha deployment/aws-mcp -- env | grep AWS

• Check all pods using the following command:

  kubectl get pods -n avesha -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'

Common Issues and Solutions

Missing Required Variables

Problem: Pod fails to start due to missing environment variables.

# Check pod events
kubectl describe pod -n avesha -l app.kubernetes.io/name=backend

# Check logs
kubectl logs -n avesha -l app.kubernetes.io/name=backend

Solution: Add missing variables to your secret.

kubectl patch secret obliq-secrets -n avesha \
  --type='json' \
  -p='[{"op": "add", "path": "/data/MISSING_VAR", "value": "'$(echo -n "your-value" | base64)'"}]'

Wrong Secret Name/Namespace

Problem: Chart can't find the specified secret.

# Check if secret exists in correct namespace
kubectl get secret obliq-secrets -n avesha

Solution: Ensure secret name matches configuration and is in correct namespace

Invalid Secret Values

Problem: Services fail to authenticate with external APIs

# Test OpenAI API
kubectl exec -n avesha deployment/backend -- curl -H "Authorization: Bearer $OPENAI_API_KEY" https://api.openai.com/v1/models

# Test AWS credentials
kubectl exec -n avesha deployment/aws-mcp -- aws sts get-caller-identity

Solution: Verify credentials are correct and have proper permissions

Pods Not Picking Up Secret Changes

Problem: Updated secret values not reflected in running pods

Solution: Restart deployments so that the updated secret values are effective.

# Restart deployments to pick up new values
kubectl rollout restart deployment -n avesha

Secret Management Best Practices

1. Use Descriptive Names: Use clear, descriptive names for your secrets
2. Organize by Environment: Create separate secrets for dev/staging/prod
3. Regular Rotation: Implement a process for regular secret rotation
4. Least Privilege: Only include the secrets each service actually needs
5. Backup Secrets: Ensure you have secure backups of critical secrets
6. Monitor Access: Use RBAC to control who can access secrets