Building a Multi-Cluster GitOps Platform with ArgoCD on GKE

This guide walks through setting up a production-grade multi-cluster GitOps platform using ArgoCD on Google Kubernetes Engine (GKE). We’ll cover the hub-and-spoke architecture, automated cluster registration, environment-specific deployments, and secret management.

Architecture Overview

Hub-and-Spoke Topology

                    ┌─────────────────────────────────────────┐
                    │           Management Cluster            │
                    │              (DevSecOps)                │
                    │  ┌─────────────────────────────────┐   │
                    │  │         ArgoCD Server           │   │
                    │  │   ┌─────────────────────────┐   │   │
                    │  │   │  ApplicationSets        │   │   │
                    │  │   │  (Multi-cluster apps)   │   │   │
                    │  │   └──────────┬──────────────┘   │   │
                    │  └──────────────┼──────────────────┘   │
                    └─────────────────┼──────────────────────┘
                                      │
           ┌──────────────────────────┼──────────────────────────┐
           │                          │                          │
           ▼                          ▼                          ▼
┌─────────────────────┐  ┌─────────────────────┐  ┌─────────────────────┐
│   Sandbox Cluster   │  │   Staging Cluster   │  │  Production Cluster │
│                     │  │                     │  │                     │
│  ┌───────────────┐  │  │  ┌───────────────┐  │  │  ┌───────────────┐  │
│  │ Kong Ingress  │  │  │  │ Kong Ingress  │  │  │  │ Kong Ingress  │  │
│  │ Cert Manager  │  │  │  │ Cert Manager  │  │  │  │ Cert Manager  │  │
│  │ ESO           │  │  │  │ ESO           │  │  │  │ ESO           │  │
│  │ Prometheus    │  │  │  │ Prometheus    │  │  │  │ Prometheus    │  │
│  │ Your Apps     │  │  │  │ Your Apps     │  │  │  │ Your Apps     │  │
│  └───────────────┘  │  │  └───────────────┘  │  │  └───────────────┘  │
│                     │  │                     │  │                     │
│   Auto-Sync: ✓      │  │   Auto-Sync: ✓      │  │   Auto-Sync: ✗      │
└─────────────────────┘  └─────────────────────┘  └─────────────────────┘

Key Principles:

• Single source of truth: All configuration lives in Git
• Environment parity: Same charts, different values per environment
• Progressive delivery: Auto-sync for non-prod, manual approval for prod
• Centralized management: One ArgoCD instance manages all clusters

Project Structure

.
├── bootstrap/
│   ├── argocd-self.yaml          # ArgoCD self-management
│   └── argocd-oidc-secrets.yaml  # SSO configuration
├── charts/
│   ├── kong-ingress/             # Wrapper chart example
│   │   ├── Chart.yaml
│   │   ├── values.yaml           # Base values
│   │   ├── values/
│   │   │   ├── sandbox.yaml      # Sandbox overrides
│   │   │   ├── staging.yaml      # Staging overrides
│   │   │   └── production.yaml   # Production overrides
│   │   └── templates/
│   ├── actions-runner-controller/
│   ├── external-secrets-operator/
│   └── kube-prometheus-stack/
├── applicationsets/
│   ├── kong-ingress.yaml         # Multi-cluster deployment
│   ├── cert-manager.yaml
│   ├── external-secrets-operator.yaml
│   └── kube-prometheus-stack.yaml
├── applications/
│   └── chartmuseum.yaml          # Single-cluster deployment
├── projects/
│   ├── cicd.yaml                 # CI/CD tools project
│   ├── monitoring.yaml           # Observability project
│   └── security.yaml             # Security tools project
└── config.yaml                   # Cluster configuration

Step 1: Cluster Configuration

Define your clusters in config.yaml:

argocd:
  namespace: argo
  hostname: argo.yourdomain.com
  chart_version: "9.1.7"

github:
  org: your-org
  repo: your-gitops-repo
  app_id: "123456"                  # GitHub App auth
  app_installation_id: "12345678"
  app_private_key_path: github-app.pem

clusters:
  - name: devops
    project: devops-project-id
    cluster: management-cluster
    location: us-central1-a
    management: true                # ArgoCD runs here

  - name: sandbox
    project: sandbox-project-id
    cluster: sandbox-cluster
    location: us-east4-a
    management: false

  - name: staging
    project: staging-project-id
    cluster: staging-cluster
    location: us-east4-a
    management: false

  - name: production
    project: production-project-id
    cluster: production-cluster
    location: us-west1-a
    management: false

Step 2: Bootstrap ArgoCD

Install ArgoCD on Management Cluster

Using Helm:

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

helm install argocd argo/argo-cd \
  --namespace argo \
  --create-namespace \
  --set server.ingress.enabled=true \
  --set server.ingress.hostname=argo.yourdomain.com \
  --set configs.params."server\.insecure"=true

Enable Self-Management

Create bootstrap/argocd-self.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd
  namespace: argo
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: cicd
  source:
    repoURL: https://github.com/your-org/your-gitops-repo.git
    targetRevision: main
    path: charts/argocd
    helm:
      releaseName: argocd
      valueFiles:
        - values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: argo
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Apply it:

kubectl apply -f bootstrap/argocd-self.yaml

Now ArgoCD manages itself via GitOps.

Step 3: Register Target Clusters

Each target cluster needs:

1. A service account with cluster-admin permissions
2. A long-lived token for ArgoCD to authenticate
3. Registration as a cluster secret in ArgoCD

Create Service Account on Target Cluster

# Switch to target cluster
kubectl config use-context <target-cluster>

# Create namespace and service account
kubectl create namespace argocd-manager
kubectl create serviceaccount argocd-manager -n argocd-manager

# Create cluster role binding
kubectl create clusterrolebinding argocd-manager \
  --clusterrole=cluster-admin \
  --serviceaccount=argocd-manager:argocd-manager

# Create long-lived token (Kubernetes 1.24+)
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: argocd-manager-token
  namespace: argocd-manager
  annotations:
    kubernetes.io/service-account.name: argocd-manager
type: kubernetes.io/service-account-token
EOF

# Get the token
TOKEN=$(kubectl get secret argocd-manager-token -n argocd-manager -o jsonpath='{.data.token}' | base64 -d)

Register Cluster in ArgoCD

# Switch back to management cluster
kubectl config use-context <management-cluster>

# Create cluster secret
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: cluster-staging
  namespace: argo
  labels:
    argocd.argoproj.io/secret-type: cluster
    environment: staging
type: Opaque
stringData:
  name: staging
  server: https://<cluster-api-server>
  config: |
    {
      "bearerToken": "${TOKEN}",
      "tlsClientConfig": {
        "insecure": false,
        "caData": "<base64-ca-cert>"
      }
    }
EOF

The environment label is crucial for ApplicationSets to target specific environments.

Step 4: Create ArgoCD Projects

Organize applications into projects with appropriate permissions:

# projects/monitoring.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: monitoring
  namespace: argo
spec:
  description: Observability and monitoring tools

  sourceRepos:
    - 'https://github.com/your-org/your-gitops-repo.git'
    - 'https://prometheus-community.github.io/helm-charts'
    - 'https://grafana.github.io/helm-charts'

  destinations:
    - namespace: monitoring
      server: '*'
    - namespace: prometheus
      server: '*'

  clusterResourceWhitelist:
    - group: ''
      kind: Namespace
    - group: 'monitoring.coreos.com'
      kind: '*'
    - group: 'apiextensions.k8s.io'
      kind: CustomResourceDefinition

  namespaceResourceWhitelist:
    - group: '*'
      kind: '*'

Step 5: The Wrapper Chart Pattern

For external Helm charts that need environment-specific values, use the wrapper chart pattern.

Structure

charts/kong-ingress/
├── Chart.yaml
├── values.yaml           # Base/shared values
├── values/
│   ├── sandbox.yaml      # Sandbox-specific
│   ├── staging.yaml      # Staging-specific
│   └── production.yaml   # Production-specific
└── templates/
    └── externalsecret.yaml  # Optional: secret management

Chart.yaml

apiVersion: v2
name: kong-ingress
description: Kong Ingress Controller wrapper chart
type: application
version: 1.0.0
appVersion: "3.0"

dependencies:
  - name: kong
    version: "2.35.0"
    repository: "https://charts.konghq.com"

values.yaml (Base)

kong:
  proxy:
    type: LoadBalancer
  ingressController:
    enabled: true
  admin:
    enabled: true

values/production.yaml (Override)

kong:
  replicaCount: 3
  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: 2000m
      memory: 2Gi
  proxy:
    annotations:
      cloud.google.com/load-balancer-type: "External"

Step 6: ApplicationSets for Multi-Cluster Deployment

ApplicationSets automatically create Applications for each matching cluster.

Basic Multi-Cluster ApplicationSet

# applicationsets/kong-ingress.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: kong-ingress
  namespace: argo
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]

  generators:
    - clusters:
        selector:
          matchLabels:
            argocd.argoproj.io/secret-type: cluster
        values:
          environment: '{{ index .metadata.labels "environment" }}'

  template:
    metadata:
      name: 'kong-ingress-{{ .name }}'
      namespace: argo
      labels:
        app.kubernetes.io/name: kong-ingress
        environment: '{{ .values.environment }}'
      finalizers:
        - resources-finalizer.argocd.argoproj.io

    spec:
      project: infrastructure

      source:
        repoURL: 'https://github.com/your-org/your-gitops-repo.git'
        targetRevision: main
        path: charts/kong-ingress
        helm:
          releaseName: kong-ingress
          valueFiles:
            - values.yaml
            - 'values/{{ .values.environment }}.yaml'

      destination:
        server: '{{ .server }}'
        namespace: kong-system

      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true
        retry:
          limit: 5
          backoff:
            duration: 5s
            factor: 2
            maxDuration: 3m

Environment-Specific Auto-Sync

Disable auto-sync for production:

generators:
  - clusters:
      selector:
        matchLabels:
          argocd.argoproj.io/secret-type: cluster
      values:
        environment: '{{ index .metadata.labels "environment" }}'
        autoSync: '{{ if eq (index .metadata.labels "environment") "production" }}false{{ else }}true{{ end }}'

template:
  spec:
    syncPolicy:
      {{ if eq .values.autoSync "true" }}
      automated:
        prune: true
        selfHeal: true
      {{ end }}

Step 7: Secret Management with External Secrets Operator

Architecture

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Google Secret  │     │  External       │     │  Kubernetes     │
│  Manager        │────▶│  Secrets        │────▶│  Secret         │
│  (per project)  │     │  Operator       │     │  (in cluster)   │
└─────────────────┘     └─────────────────┘     └─────────────────┘

Deploy ESO to All Clusters

# applicationsets/external-secrets-operator.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: external-secrets-operator
  namespace: argo
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            argocd.argoproj.io/secret-type: cluster

  template:
    metadata:
      name: 'external-secrets-{{ .name }}'
    spec:
      project: security
      source:
        repoURL: https://charts.external-secrets.io
        chart: external-secrets
        targetRevision: 0.9.11
        helm:
          valuesObject:
            installCRDs: true
            serviceAccount:
              annotations:
                iam.gke.io/gcp-service-account: eso@{{ .name }}-project.iam.gserviceaccount.com
      destination:
        server: '{{ .server }}'
        namespace: external-secrets

Create ClusterSecretStore

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: gcpsm-secret-store
spec:
  provider:
    gcpsm:
      projectID: <gcp-project-id>
      auth:
        workloadIdentity:
          clusterLocation: <location>
          clusterName: <cluster-name>
          clusterProjectID: <project-id>
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets

Reference Secrets in Charts

# templates/externalsecret.yaml
{{- if .Values.externalSecrets.enabled }}
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: {{ .Release.Name }}-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcpsm-secret-store
    kind: ClusterSecretStore
  target:
    name: {{ .Release.Name }}-secrets
  dataFrom:
    - extract:
        key: {{ .Values.externalSecrets.secretName }}
{{- end }}

Step 8: Handling Sync Differences

ArgoCD may show “OutOfSync” for fields that Kubernetes mutates. Use ignoreDifferences:

spec:
  ignoreDifferences:
    # Ignore cluster-assigned IPs
    - group: ""
      kind: Service
      jsonPointers:
        - /spec/clusterIP
        - /spec/clusterIPs

    # Ignore HPA-managed replicas
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

    # Ignore auto-generated CRD fields
    - group: apiextensions.k8s.io
      kind: CustomResourceDefinition
      jsonPointers:
        - /spec/preserveUnknownFields

    # Use jqPathExpressions for complex patterns
    - group: monitoring.coreos.com
      kind: ServiceMonitor
      jqPathExpressions:
        - .spec.endpoints[]?.metricRelabelings

  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true

Step 9: Notifications

Configure Slack notifications for sync events:

# In argocd-notifications-cm ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argo
data:
  service.slack: |
        token: $slack-token

  template.app-sync-succeeded: |
    message: |
      Application {{.app.metadata.name}} sync succeeded.
      Revision: {{.app.status.sync.revision}}    

  template.app-sync-failed: |
    message: |
      Application {{.app.metadata.name}} sync failed!
      Error: {{.app.status.operationState.message}}    

  trigger.on-sync-succeeded: |
    - when: app.status.operationState.phase == 'Succeeded'
      send: [app-sync-succeeded]    

  trigger.on-sync-failed: |
    - when: app.status.operationState.phase == 'Failed'
      send: [app-sync-failed]    

Add annotations to ApplicationSets:

template:
  metadata:
    annotations:
      notifications.argoproj.io/subscribe.on-sync-succeeded.slack: deployments
      notifications.argoproj.io/subscribe.on-sync-failed.slack: alerts

Step 10: Day 2 Operations

Adding a New Cluster

1. Add to config.yaml:

   clusters:
     - name: new-cluster
       project: new-project-id
       cluster: new-cluster-name
       location: us-east4-a
       management: false
   

2. Create service account and register:

   argocd-setup add-cluster new-cluster
   

3. Label the cluster secret:

   kubectl label secret cluster-new-cluster -n argo environment=production
   

4. ApplicationSets automatically deploy to new cluster.

Upgrading ArgoCD

1. Update chart version:

   # charts/argocd/Chart.yaml
   dependencies:
     - name: argo-cd
       version: "6.0.0"  # New version
   

2. Commit and push.

3. ArgoCD syncs itself automatically.

Adding a New Chart

1. Create wrapper chart:

   mkdir -p charts/new-app/{templates,values}
   

2. Create Chart.yaml, values.yaml, and environment overrides.

3. Create ApplicationSet:

   cp applicationsets/template.yaml applicationsets/new-app.yaml
   # Edit with appropriate values
   

4. Apply:

   kubectl apply -f applicationsets/new-app.yaml
   

5. Commit everything to Git.

Best Practices Summary

Troubleshooting

Application Stuck in “Progressing”

argocd app get <app-name> --hard-refresh
kubectl describe application <app-name> -n argo

Sync Fails with “already exists”

Enable server-side apply:

syncPolicy:
  syncOptions:
    - ServerSideApply=true

Cluster Unreachable

Check cluster secret:

kubectl get secret cluster-<name> -n argo -o yaml
argocd cluster list

ExternalSecret Not Syncing

Check ESO logs:

kubectl logs -n external-secrets deployment/external-secrets
kubectl describe externalsecret <name> -n <namespace>

Conclusion

A well-architected multi-cluster GitOps platform provides:

• Consistency: Same process for all environments
• Auditability: Git history as audit log
• Scalability: Add clusters without changing workflows
• Security: No credentials in Git, controlled access
• Reliability: Self-healing and automatic recovery

The initial investment in setting up this infrastructure pays dividends as your organization scales. Every new application, every new cluster, every new team benefits from the established patterns.