Static Manifest Propagation From Seed To Shoots

Overview

Static manifest propagation is a mechanism that allows operators to distribute predefined Kubernetes resources across all Shoot clusters automatically. By placing labeled Secrets in the seed cluster's garden namespace, operators can ensure that specific manifests (such as RBAC rules, quotas, config policies, or compliance objects) are consistently deployed to every Shoot cluster without manual intervention.

Why Use Static Manifests?

Static manifest propagation provides several benefits for cluster operators:

• Centralized Management: Update manifests in one location (the seed's garden namespace) and have changes automatically propagate to all Shoots
• Consistency: Ensure all Shoot clusters have the same baseline configurations, policies, or resources
• Simplified Operations: Eliminate the need for manual per-Shoot provisioning or custom controllers
• Generic Distribution: Works independently of cloud provider or extension logic
• Compliance & Governance: Easily enforce organization-wide policies, quotas, or security configurations across all clusters

Common use cases include:

• Deploying RBAC rules or ClusterRoles
• Setting ResourceQuotas or LimitRanges
• Distributing NetworkPolicys
• Injecting compliance or audit configurations
• Providing common ConfigMaps or monitoring agents

How It Works

During Shoot reconciliation, the gardenlet performs the following steps:

1. Scans the seed cluster's garden namespace for Secrets labeled with gardener.cloud/purpose=shoot-static-manifest.
2. Copies all matching Secrets into each Shoot namespace.
3. Creates a single ManagedResource that references these Secrets.
4. The ManagedResource ensures the manifests are applied to the Shoot cluster.

This process happens automatically during every Shoot reconciliation, ensuring manifests stay synchronized.

How to Propagate Static Manifests

Step 1: Prepare Your Manifests

Create a YAML file containing the Kubernetes resources you want to deploy to all Shoot clusters. For example:

# my-manifests.yaml
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: default-quota
  namespace: default
spec:
  hard:
    requests.cpu: "100"
    requests.memory: 200Gi
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: org-viewer
rules:
- apiGroups: [""]
  resources: ["pods", "services"]
  verbs: ["get", "list", "watch"]

Step 2: Create a Secret in the Seed Cluster

Create a Secret in the seed cluster's garden namespace containing your manifests. The Secret must be labeled with gardener.cloud/purpose=shoot-static-manifest.

# Create the Secret with the required label
kubectl create secret generic my-static-manifests \
  --from-file=manifests.yaml=my-manifests.yaml \
  --namespace=garden \
  --dry-run=client -o yaml | \
  kubectl label --local -f - gardener.cloud/purpose=shoot-static-manifest --dry-run=client -o yaml | \
  kubectl apply -f -

Or create it declaratively:

apiVersion: v1
kind: Secret
metadata:
  name: my-static-manifests
  namespace: garden
  labels:
    gardener.cloud/purpose: shoot-static-manifest
type: Opaque
data:
  manifests.yaml: <base64-encoded-yaml-content>

Optional: Target Specific Shoots with a Selector

By default, manifests are propagated to all Shoot clusters running on the seed. To target only specific Shoots, add the static-manifests.shoot.gardener.cloud/selector annotation with a JSON-encoded metav1.LabelSelector:

apiVersion: v1
kind: Secret
metadata:
  name: production-manifests
  namespace: garden
  labels:
    gardener.cloud/purpose: shoot-static-manifest
  annotations:
    static-manifests.shoot.gardener.cloud/selector: |
      {"matchLabels":{"environment":"production"}}
type: Opaque
data:
  manifests.yaml: <base64-encoded-yaml-content>

The annotation value must be a valid JSON representation of a Kubernetes metav1.LabelSelector that matches against Shoot labels.

Examples:

Simple label matching:

static-manifests.shoot.gardener.cloud/selector: |
  {"matchLabels":{"environment":"production"}}

Multiple labels (AND logic):

static-manifests.shoot.gardener.cloud/selector: |
  {"matchLabels":{"environment":"production","region":"us-east"}}

Match expressions (advanced selectors):

static-manifests.shoot.gardener.cloud/selector: |
  {"matchExpressions":[{"key":"environment","operator":"In","values":["production","staging"]}]}

Combining matchLabels and matchExpressions:

static-manifests.shoot.gardener.cloud/selector: |
  {
    "matchLabels":{"team":"platform"},
    "matchExpressions":[{"key":"environment","operator":"NotIn","values":["development"]}]
  }

If the selector annotation contains invalid JSON or cannot be parsed as a valid metav1.LabelSelector, the Secret will be skipped and an error will be logged.

Step 3: Verify Propagation

After the next Shoot reconciliation, verify that the manifests have been propagated:

1. Check the Shoot namespace in the seed cluster for the copied Secret:

   kubectl get secret static-manifests-my-static-manifests -n shoot--<project>--<shoot>

2. Check the ManagedResource referencing your Secret:

   kubectl get managedresource static-manifests-from-seed -n shoot--<project>--<shoot>

3. Check the Shoot cluster to confirm resources are applied:

   # Using the Shoot cluster kubeconfig
   kubectl get resourcequota default-quota -n default
   kubectl get clusterrole org-viewer

Updating Static Manifests

To update manifests across all Shoot clusters:

1. Update the Secret in the seed's garden namespace with the new manifest content.
2. Wait for the next Shoot reconciliation cycle, or manually trigger reconciliation.
3. The gardenlet will detect the change and update the ManagedResource in each Shoot namespace.
4. The updated manifests will be applied to all Shoot clusters.

Removing Static Manifests

To stop propagating manifests to Shoot clusters:

1. Delete the Secret from the seed's garden namespace:

   kubectl delete secret my-static-manifests -n garden

2. During the next reconciliation, the gardenlet will remove the Secret from Shoot namespaces.

3. The associated resources will be deleted from the Shoot clusters via the ManagedResource cleanup.

Important Considerations

• No Templating or Dynamic Logic: Manifests must be completely static. No templating, variable substitution, or dynamic logic is supported. The same exact manifests are deployed to all Shoot clusters without modification. If you need per-Shoot customization, Shoot-specific values, or sophisticated logic (e.g., conditional deployment, templating based on Shoot properties), you must write a Gardener extension instead.
• Shoot Selector: Use the static-manifests.shoot.gardener.cloud/selector annotation to target specific Shoots based on their labels. Without this annotation, manifests are propagated to all Shoots. Invalid selectors will cause the Secret to be skipped with an error logged.
• Namespace Scoping: Ensure manifests use appropriate namespaces. Resources without a namespace will be created in the default namespace of the Shoot cluster.
• Resource Conflicts: Avoid creating resources that might conflict with Gardener-managed resources or Shoot-specific configurations.
• Secret Naming: Use descriptive names for Secrets to distinguish between different sets of manifests.
• Multiple Secrets: You can create multiple labeled Secrets in the garden namespace; all will be propagated (subject to their selectors).
• Label Requirement: The label gardener.cloud/purpose=shoot-static-manifest is mandatory. Secrets without this label will not be propagated.
• Reconciliation Timing: Changes may take time to propagate depending on the Shoot reconciliation schedule.
• Health Checks: Failed resources in static manifests are propagated to the Shoot's SystemComponentsHealthy condition, allowing visibility into deployment issues.