Argo CD at 34 Clusters: GitOps Design Lessons for Multi-Cloud Kubernetes

At fleet scale, GitOps becomes a way to manage infrastructure itself, and the control plane’s failure modes matter as much as its deployment model. At Lucid Motors, one Argo CD instance managed 34 Kubernetes clusters across 22 AWS accounts and an Oracle Cloud tenancy, spanning five regions on three continents. A single repository held 174 Helm charts that rendered 1,214 Argo CD Application objects. The experience points to a practical lesson: the system’s reliability depends on clear boundaries, repeatable validation and safeguards against changes with a large blast radius.

That dependence extends to the services GitOps relies on. On Oct. 6, 2026, Red Hat published a walkthrough of using Argo CD ApplicationSets to provision hosted control planes from Git. The next day, a GitHub outage disrupted Actions, Git operations and pull requests, following smaller incidents on Oct. 5 and 6. Those events underline a tension in current GitOps practice: teams can now manage clusters declaratively, but a break in the Git layer can halt deployments and configuration changes.

Keep the hub powerful and the spokes in control

The deployment used a hub-and-spoke design. The hub contained the sole Argo CD instance for platform components, cluster registrations, the Cluster API controller identity and three ApplicationSets. One set deployed each cluster’s baseline through an app-of-apps chart. Another sent shared platform services to the clusters that hosted them. The third held the Cluster API definitions used to create clusters.

The spokes retained authority over access within their own boundaries. Each cluster’s authentication configuration determined whether it accepted the hub’s role. Each spoke also owned the role Cluster API used inside that cluster, while workloads received cloud roles through the spoke’s identity provider. That division matters: central management does not have to mean that all credentials and permissions live in one place.

On AWS, the hub held no long-lived cross-account secrets. It obtained short-lived credentials, and each access grant was declared and reviewed in Git on the spoke side. Oracle Cloud did not support the same setup in this deployment; registering a cluster there still required a credential stored in a vault. That exception is instructive. A multi-cloud design should document where its security properties differ by provider, because a shared management model can conceal different credential arrangements.

A small cluster contract drives the fleet

The central design choice was to describe each cluster in a compact configuration file of about 20 keys. Those values captured its identity, account boundary, region, environment, API endpoint access, delegated DNS and certificates, and any opt-outs. The platform derived the rest from that contract: which Applications to create, IAM role names, Cluster API identity, DNS delegation and cost tags.

That approach turns fleet changes into edits to shared inputs. Adding a cluster to a fleet meant changing a label on its registration; including a component in a fleet took one line in a list. Matrix generators handled the expansion across clusters. Instead of maintaining separate configuration for each downstream resource, the team made those resources consequences of a single cluster definition.

The contract also had to work beyond the original environment. A Middle East expansion brought an Oracle Cloud tenancy into the fleet, using the same configuration model rather than a bespoke process. That is a useful test for platform abstractions: they need to accommodate a new region or provider without multiplying one-off operational paths.

Validate rendered output, not just templates

With 174 charts, the operational risk was a values change that appeared valid for one cluster but produced a bad result elsewhere. The team addressed that risk by making CI render the full configuration graph on every merge request.

Checks covered individual charts, app-of-apps charts rendered against every cluster contract, cluster definitions, and each chart referenced by an Application using the exact values it would receive. CI also rendered the target branch and attached a manifest diff to the merge request. Reviewers could inspect the changes the clusters would receive, rather than infer the outcome from template edits.

That distinction grows more useful as a fleet expands. A template can be syntactically sound while its rendered output changes resources across many environments. Rendering against every contract gives reviewers a view of those effects before deployment. It also makes the configuration repository a place to catch incompatible changes, rather than simply a record of what has already been applied.

Make destructive actions deliberate

The team treated an unintended deletion as a leading failure mode. Every ApplicationSet preserved deployed resources when the set was deleted, preventing removal of an entry from cascading into deletion of its resources. Cluster objects also carried a no-prune setting, so reverting a commit would not remove a running cluster.

These safeguards shape how reconciliation behaves when configuration changes or disappears. The operating principle is to match the controller’s authority to the impact and reversibility of an action. That matters especially when a single hub can affect dozens of clusters: a default that seems harmless in a small setup can have fleet-wide consequences.

The same design should account for Git or hub outages. Because workloads run in the spokes, existing services can continue operating if the central control plane or Git provider becomes unavailable. Deployments and configuration updates will pause, but that interruption need not become a workload outage. The hub also deserves treatment as a highly privileged system: a compromised repository or CI path can gain the hub’s reach once a change is merged.

Track outcomes with comparable measurements

After cutover, reported out-of-band drift fell from 24.3 incidents per month to 0.9 across two 90-day windows. The figures came from different measurement methods. Before cutover, the count used incident tickets tagged as drift, and tagging was incomplete. Afterward, it used controller diff events confirmed to have an out-of-band cause. The results therefore don’t establish a clean percentage reduction.

They do show that the instrumented system saw little confirmed drift and that the controller corrected each observed case within its reconciliation interval, without manual intervention. The broader operational lesson is to use the same definition and collection method before and after a change wherever possible. Better measurement can make a system’s behavior visible, but inconsistent methods limit what a comparison can establish.

Onboard through the same path

Adding a cluster followed four steps: create IAM roles in Terraform using the established naming pattern, write the cluster values file, submit a merge request for CI to render the full graph and post its diff, then merge and label the registration. Clusters originally built by hand could be adopted in place through an explicit import path.

Mean onboarding time moved from 4.0 days across eight earlier clusters to 1.7 hours across 11 clusters after cutover. The first figure was reconstructed from change tickets; the second measured time from merge to first successful sync. The cutover also removed manual approval steps, and the windows and samples differed, so the numbers aren’t a direct comparison. They still indicate that a defined contract and automated rendering can shorten the path from cluster configuration to managed state.

Scale doesn’t change the basic GitOps loop: declare the desired state, review it and let a controller reconcile the difference. It raises the cost of shortcuts. A credential in the hub, an unrendered values change or a default pruning rule can affect a much larger fleet. The practical takeaway is to make cluster configuration the shared contract, test the rendered result across that contract, and build deletion and outage behavior into the control plane before the fleet grows.