Stop Kubernetes Admission Control Failing Open: 11 Tips

Short version: an admission webhook with failurePolicy: Ignore lets the request through when it times out, and the API server records that in apiserver_admission_webhook_fail_open_count instead of denying anything. To prove enforcement you need three artifacts: the list of every Ignore webhook with its timeout, an alert on that counter using type="admit" for mutating webhooks, and the CEL-expressible rules moved into ValidatingAdmissionPolicy where there is no network call to lose.

The reader I have in mind is the one who has to put the answer in writing. A control-effectiveness questionnaire asks whether privileged pods can be admitted. An SRE has to explain a pod that exists in a namespace where policy said it could not. The usual answer is a screenshot of Kyverno or Gatekeeper in Enforce mode, which proves the policy object exists and says nothing about the seconds when the webhook behind it was unreachable. Admission control is also the wrong layer for a whole class of questions, so pair it with controls that keep working when the API server is busy: a default-deny egress policy stops pod exfiltration regardless of what got admitted, and Kubernetes v1.37 tightened the control plane itself, which is why mirror pods may no longer reference secrets.

The path a request takes, and the branch that produces a written object with no denial:

Before the list, the mapping worth keeping next to your policy repo. Rows where the evidence column reads "the policy exists" are the expensive ones.

Eleven checks, and what each one proves

Tips 1 to 4 establish what is enforced right now. Tips 5 to 8 name what each control refuses to answer. Tips 9 to 11 cover what survives a control-plane restart.

1. Inventory every webhook permitted to fail open, with its timeout, before claiming anything is enforced. The v1 API defaults failurePolicy to Fail, so every Ignore in your cluster was chosen by a person or a Helm chart. Service-mesh injectors, certificate controllers and image-mutation sidecars ship Ignore deliberately to avoid wedging the cluster, and that choice is defensible until the same configuration also carries your security rules. Every cluster I have run this against had at least one Ignore nobody present remembered choosing:

   kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o json \
     | jq -r '.items[] | .metadata.name as $c | .webhooks[]
              | select(.failurePolicy == "Ignore")
              | [$c, .name, (.timeoutSeconds // 10)] | @tsv'

1. Alert on apiserver_admission_webhook_fail_open_count, and use type="admit" for mutating webhooks. The counter increments each time a request is allowed through because the webhook errored while its policy said Ignore. Kubernetes PR #127898 corrected the help text in October 2024: the type label carries admit or validating, where the documentation had long promised mutating. A dashboard filtered on type="mutating" returns an empty series forever and reads as a clean bill of health, which is the worst possible failure mode for a control you are attesting to.

   sum by (name, type) (rate(apiserver_admission_webhook_fail_open_count[5m]))

1. Treat a fail-open spike as a signal to investigate, and know the one thing that inflates it. kubernetes/kubernetes#118829 documents a client cancelling a request mid-webhook-call being counted as a fail-open event. Somebody hitting Ctrl-C on a slow kubectl apply therefore moves the same counter that a real outage moves. Correlate against apiserver_admission_webhook_request_total and the webhook pod's own readiness before you open an incident ticket.
2. Move every rule CEL can express into ValidatingAdmissionPolicy, because an in-process check has no network to lose. VAP evaluates inside kube-apiserver, which shrinks the failure modes to CEL compile errors and runtime errors, and its failurePolicy governs only those (Kubernetes docs). Start with the rules that read a single object: privileged containers, hostNetwork, missing resource limits, forbidden image registries. Those are the four that show up in questionnaires and they need no external state.
3. Run validationActions: [Audit] first, then hunt the annotation the audit log actually writes. Failures in Audit mode land in the audit event under the key validation.policy.admission.k8s.io/validation_failure, and that annotation is the queryable artifact you hand over as evidence of coverage before flipping to Deny. The API rejects Deny and Warn together, so the useful pre-production pairing is [Warn, Audit]: authors see the message, you keep the record.

   jq -r 'select(.annotations["validation.policy.admission.k8s.io/validation_failure"])
          | [.requestReceivedTimestamp, .user.username, .objectRef.namespace,
             .annotations["validation.policy.admission.k8s.io/validation_failure"]] | @tsv' audit.log

1. Write down what VAP is silent on in the same document that claims the win. CEL in the API server makes no external calls and holds no state, so image signature verification, cross-resource lookups beyond a paramKind, resource generation and cleanup all stay with Kyverno or Gatekeeper. So does the category of resources that already exist: admission control has never evaluated an object created before the policy was written, which is what background scanning covers, and what runtime detection covers after that. Deciding which sensor owns the post-admission half is its own exercise, and I have argued the trade-offs in Falco vs Tetragon vs Tracee.
2. On Gatekeeper v3.20 or later you have two enforcement points, so confirm they share a scope. VAP generation reached beta and default-on in v3.20 with --default-create-vap-for-templates and --default-create-vap-binding-for-constraints both true (Gatekeeper docs). The sharp edge: a VAP is generated only for ConstraintTemplates that carry the K8sNativeValidation CEL engine, engine priority is fixed, and there is no fallback to Rego. A Rego-only template keeps running purely on the webhook while your dashboard reports VAP enabled.

   kubectl get constrainttemplates -o json \
     | jq -r '.items[] | [.metadata.name,
         ((.spec.targets[0].code // []) | map(.engine) | join(",") | if . == "" then "rego-only" else . end)] | @tsv'

1. Design around the shape of ApplyConfiguration before you retire a mutating webhook. MutatingAdmissionPolicy has been stable since Kubernetes v1.36, and its ApplyConfiguration patches use server-side apply semantics, which forbids modifying atomic structs, maps or arrays (Kubernetes docs). Surgical edits to fields Kubernetes marks atomic need JSONPatch instead. Set reinvocationPolicy: IfNeeded where your policy has to re-run after another policy mutates the object, otherwise ordering decides your outcome.
2. Prove that nobody outside the platform team can delete the enforcement. A ValidatingAdmissionPolicyBinding is an ordinary API object: delete the binding and the policy stops applying, with no denial, no warning, and one audit entry that looks like housekeeping. Enumerate who holds delete on the admissionregistration.k8s.io group, wildcards included, then check which of those subjects are still authenticating with long-lived credentials (the same argument as killing static ServiceAccount tokens).

   kubectl get clusterroles -o json | jq -r '
     .items[] | .metadata.name as $r | .rules[]?
     | select((.apiGroups[]? | . == "admissionregistration.k8s.io" or . == "*")
           and (.verbs[]? | . == "delete" or . == "*")) | $r' | sort -u

1. Close the bootstrap and self-protection gaps with manifest-based admission config, beta in v1.37. KEP-5793 states that file-configured admission policies and webhooks must be active before the API server begins processing requests, and that manifest-based controls cannot be bypassed or modified through the Kubernetes API. Objects loaded this way carry a .static.k8s.io name suffix, and an invalid manifest stops the API server reaching ready, which is a loud failure by design. Kubernetes v1.37 (Garhwal) was released on 26 August 2026, so treat this as a control-plane change with a rollback plan, tested on one control-plane node at a time.

    apiVersion: apiserver.config.k8s.io/v1
    kind: AdmissionConfiguration
    staticManifestsDir: /etc/kubernetes/admission-static

1. Build the coverage matrix, one row per question you will be asked. Use the five columns in the table above: the control question, the mechanism, the enforcement point, the queryable evidence, and the window where it is blind. Filling it in takes a half day of reading manifests and produces a document that outlives every dashboard in this list, because the column that matters is the blind window and no tool populates it for you.

Wrap-up

The habit that carries the rest: never accept "the policy is in Enforce mode" as evidence. Ask for the artifact, meaning the metric series, the audit annotation, or the RBAC list, and where no artifact exists, record the blind window in place of the reassurance.

The counterargument is worth airtime. Teams running failurePolicy: Fail on every webhook have paid for it with a cluster that refused all writes when the policy pod crash-looped, and Ignore on the injector is how they stopped that recurring. That position buys availability and costs the ability to answer question one. Pick deliberately, write down which you picked, and date it.

Tips 1 through 5 fit inside a week with the staff you have. Tips 7 through 10 touch the control plane, need a maintenance window, and need an owner for the evidence trail afterwards. When that second half keeps slipping, a scoped admission-control review is the usual way in: inventory every webhook and policy, separate the rules CEL can hold from the rules that need an external engine, produce the audit queries your control-effectiveness answers point at, and hand back the blind-window list with a cost against each fix. Somebody still has to approve the API server restart.