GEP-5093: Gateway Address Routability

  • Issue: #5093
  • Status: Implementable

(See status definitions.)

This GEP obsoletes GEP-1651: Gateway Routability. See Background for the relationship and prior iterations.

TLDR

Add a routability field to Gateway addresses (spec.addresses and status.addresses) so that users can request, and implementations can report, the reachability scope of each address a Gateway uses.

Motivation

Gateway API currently treats all addresses as opaque values with a type (IPAddress, Hostname) but no indication of where those addresses are reachable from. There is no portable way to say “give me an address that is only reachable inside the cluster” or to discover whether a provisioned address has the Cluster guarantee, an implementation-specific scope, or no portable guarantee.

This gap blocks several use cases:

  • Internal-only gateways. Knative and similar projects need to deploy Gateways that are reachable within the cluster but not from the public internet (#1651). Today this requires implementation-specific annotations or out-of-band Service manipulation.

  • Egress gateways. Workloads that route outbound traffic through a Gateway need a cluster-internal address to connect to. Without a portable way to request or identify such an address, egress patterns cannot be standardized. Standardizing such patterns has been requested by the wg-ai-gateway in service of generative AI use cases: for example, a Gateway with a Cluster scoped address and a Backend pointing to an external inference provider should generally not be reachable from the open internet, to avoid injecting inference credentials into arbitrary requests. (see #4746 discussions on “open relays”.)

  • Multi-address gateways. A Gateway may be provisioned with both a public and an internal address. Clients currently have no way to determine which address is appropriate for their context.

Background

This GEP has two lines of origin.

GEP-1651: Gateway-level routability

This GEP obsoletes GEP-1651, which proposed a routability field under spec.infrastructure.routability on the Gateway. That design treated routability as a Gateway-level concern: one scope per Gateway, drawn from a Public / Private / Cluster enum, with status addresses required to be semantically no wider than the requested scope.

GEP-1651 correctly identified the problem and Cluster is preserved here. Its “Alternatives” section even anticipated per-address routability and recorded the trade-offs. We are indebted to dprotaso and sunjayBhatia for the foundational work.

GEP-1651 did not progress past Provisional. The blocking concerns were:

  • Private needs decomposition. It was defined as “routable inside a private network larger than a single cluster (e.g. VPC) and MAY include RFC1918 address space.” Reviewer comments reveal a need to decompose Private into two distinct scopes–a VPC-internal address and a cluster-internal address–with different operational implications (whether kube-proxy captures traffic, whether the address is reachable from adjacent clusters). Without that decomposition, a precise definition could not be agreed.
  • Multi-network Kubernetes. Concurrent work on multi-network support upstream (KEP-3700) made any binary classification of “private” feel premature, because the boundary of “the private network” is itself becoming an administrator-defined concept.
  • Single scope per Gateway is too coarse. Real deployments provision Gateways with both a public and a cluster-internal address (e.g. an ingress Gateway also reachable from inside the cluster for mesh-internal callers). A Gateway-level field cannot express this.
  • No portable per-address discovery. Consumers (workloads, agents, egress tooling) had no reliable way to read the reachability of an individual address from status.addresses, because the scope lived on the Gateway rather than on each address.

GEP-4747: L7 reverse-proxy egress

The second line of origin is GEP-4747 (PR #4746), which proposed L7 reverse-proxy egress using the existing Gateway resource. During review, a portable way to request a cluster-internal address was listed by Gateway API maintainers as the GEP’s “biggest requirement”–agents need an IP to connect to and must be able to programmatically determine its reachability from the Gateway’s status (#4746 comment).

An initial attempt to absorb that requirement into GEP-4747 (by adding ClusterIP as an address type and a Gateway type field) caused the GEP to accumulate unrelated debates–TLS policy scoping, open-relay prevention, ingress/egress intent encoding–that were valuable in their own right but not load-bearing for the core egress model. At howardjohn’s suggestion (review), the GEP was closed and split into smaller proposals, of which this is the first (closing comment).

The split was deliberate: reachability is a prerequisite for egress, but its scope is broader than egress alone (it also covers internal-only ingress, multi-address Gateways, and discovery by arbitrary consumers). Pursuing it here lets GEP-4747 and any companion proposals reference a settled reachability model rather than re-litigating it.

User Stories

  • As a platform operator, I want to request a cluster-internal address for a Gateway so that it is not exposed to the public internet.
  • As a workload developer, I want to discover from a Gateway’s status whether its address has the Cluster guarantee, an implementation-defined scope (e.g., a VPC), or no portable reachability guarantee.
  • As an implementation author, I want to express implementation-specific routability scopes without waiting for upstream API changes.
  • As a workload developer, I want to direct traffic at a suitable (ideally cluster-internal) Gateway address without consuming the status subobject, e.g. by targeting a Service or an EndpointSelector. (Out of scope for this GEP; see Non-Goals and KEP-6116.)

Goals

  • Define a routability field on GatewaySpecAddress and GatewayStatusAddress with well-known values that cover the most common scopes.
  • Allow implementations to report routability in status even when the user did not request a specific scope in spec.
  • Support prefixed custom routability values for implementation-specific scopes.

Non-Goals

  • Defining enforcement mechanisms (e.g. NetworkPolicy) for restricting traffic to or from a Gateway.
  • Validating actual reachability other than the ServiceCIDR and in-cluster reachability requirements for Cluster IPAddress values.
  • Exposing Service-level fields (loadBalancerClass, sessionAffinity, etc.) on Gateway. Those concerns belong in a separate effort.
  • Recording or enforcing Gateway intent (e.g. an ingress/egress type field). This GEP defines reachability of addresses; it takes no opinion on whether a Gateway is intended for ingress, egress, or both, or on how that intent is recorded or enforced. That is pursued in a separate GEP split from #4746.
  • Providing a way to direct traffic to a Gateway’s internal addresses without reading status, e.g. via a Service or EndpointSelector. This is a natural follow-on for discovery ergonomics and is deferred to KEP-6116 rather than pursued here.

API

Routability Field

A new optional routability field is added to both GatewaySpecAddress and GatewayStatusAddress. It uses a new GatewayAddressRoutabilityType string type:

// GatewayAddressRoutabilityType describes where a Gateway address is expected to
// be reachable from.
//
// Valid values are empty, `Cluster`, or a prefixed implementation-specific value.
// The `k8s.io` domain and all its subdomains are reserved and cannot be used
// until Gateway API defines a value for them.
//
// <gateway:experimental:validation:MaxLength=253>
// Prefixed values require a valid DNS subdomain prefix and a non-empty path.
// <gateway:experimental:validation:XValidation:message="Routability must be empty, Cluster, or an implementation-specific prefixed path; k8s.io and its subdomains are reserved",rule="size(self) == 0 || self == 'Cluster' || (self.matches('^.*/.+$') && !format.dns1123Subdomain().validate(self.split('/')[0]).hasValue() && self.split('/')[0] != 'k8s.io' && !self.split('/')[0].endsWith('.k8s.io'))">
type GatewayAddressRoutabilityType string

const (
	// GatewayAddressRoutabilityDefault uses the implementation's default
	// address provisioning behavior.
	GatewayAddressRoutabilityDefault GatewayAddressRoutabilityType = ""

	// GatewayAddressRoutabilityCluster indicates that an IPAddress is a
	// Kubernetes Service ClusterIP.
	GatewayAddressRoutabilityCluster GatewayAddressRoutabilityType = "Cluster"
)

type GatewaySpecAddress struct {
	// Existing fields omitted.

	// Routability specifies the requested reachability scope of this address.
	// Valid values are empty, `Cluster`, or a prefixed implementation-specific value.
	// The `k8s.io` domain and all its subdomains are reserved and cannot be used
	// until Gateway API defines a value for them. When unset or empty, this field uses the
	// implementation's default routability behavior.
	// Support: Extended
	//
	// <gateway:util:excludeFromCRD>
	// Notes for implementors:
	//
	// Implementations claiming GatewayAddressRoutability MUST report routability
	// for every resulting status address.
	// </gateway:util:excludeFromCRD>
	//
	// +optional
	// <gateway:experimental>
	Routability GatewayAddressRoutabilityType `json:"routability,omitempty,omitzero"`
}

type GatewayStatusAddress struct {
	// Existing fields omitted.

	// Routability reports the reachability scope of this address. When empty or
	// unset, this field uses the implementation's default routability behavior.
	//
	// <gateway:util:excludeFromCRD>
	// Notes for implementors:
	//
	// Implementations claiming GatewayAddressRoutability MUST set this field
	// for every status address.
	// </gateway:util:excludeFromCRD>
	//
	// +optional
	// <gateway:experimental>
	Routability *GatewayAddressRoutabilityType `json:"routability,omitempty"`
}

CRD validation MUST accept empty, Cluster, and implementation-specific values with a valid DNS subdomain prefix and a non-empty path. It MUST reject other unprefixed values, invalid prefixes, and values whose DNS subdomain prefix is k8s.io or ends in .k8s.io. Prefixed values are implementation-specific: a valid value is not necessarily supported by every implementation.

Well-Known Values

The set of well-known values is intentionally open-ended. Because the field is a string and consumers must already tolerate values they do not recognize (including prefixed ones), new well-known scopes can be added in future revisions without breaking compatibility. Adding one requires updating the CRD validation.

The default and Cluster model below is a portable starting point, not a ceiling: if experience (e.g. multi-network Kubernetes, KEP-3700) or expansion of LoadBalancer semantics (KEP-6128) shows that additional scopes are needed, they can be introduced without disrupting existing Gateways.

  • Empty: An omitted or empty value uses the implementation’s default routability behavior without imposing a routability requirement. In status, an implementation claiming GatewayAddressRoutability MUST explicitly report an empty value for an address without another reported routability value. The empty value has no portable reachability guarantee and no ServiceCIDR validation applies.

  • Cluster: For IPAddress addresses, the reported address MUST be a ClusterIP in Kubernetes Service terms: it MUST be in the cluster’s ServiceCIDR and MUST be reachable from within the cluster. It MAY be routable outside the cluster at the network administrator’s discretion. It SHOULD use a non-globally-routable address (for example, RFC 1918 or RFC 4193) unless the cluster, including its ServiceCIDR, uses globally routable addresses.

The field also accepts prefixed values (for example, example.com/CorpWan or example.com/PublicVPC) for implementation-specific scopes or internal address ranges (RFC 1918, RFC 4193, RFC 6598). The prefix must be a valid DNS subdomain. These values have no portability guarantee and are defined by the implementation that supports them. Prefixes equal to k8s.io or ending in .k8s.io are reserved until Gateway API defines a corresponding well-known value.

testing.x-k8s.io/sentinel is reserved for conformance. It carries no routability guarantee. Implementations claiming GatewayAddressRoutability MUST support and report it for the conformance request. Other values using the testing.x-k8s.io prefix are reserved and MUST be treated as unsupported. This reservation is semantic only; CRD validation intentionally does not special-case it.

Spec Semantics

When routability is set to Cluster or a prefixed value in spec.addresses, it forms a requirement on any address the implementation provisions for that entry–the same way specifying an exact value does. When unset or empty, it uses the implementation’s default routability behavior and imposes no routability requirement.

For an IPAddress result, Cluster MUST use a ClusterIP from the ServiceCIDR.

An implementation MUST NOT satisfy a Cluster or prefixed entry with an address of a different routability value and MUST treat an unrecognized routability value as unsatisfiable.

If a requested routability cannot be satisfied, the correct behavior is to leave that entry unsatisfied and report it. Implementations MUST NOT substitute a different scope.

spec.addresses MAY contain entries with different routability values and may combine them with requests for specific addresses. Implementations MUST evaluate each address request separately.

When spec.addresses is nonempty, status.addresses MUST contain exactly one distinct matching address for every satisfied spec entry. It MUST NOT contain an address that does not match a spec entry, except an address that remains active after an update that the implementation cannot apply. A retained address MUST NOT count as satisfaction of a spec entry and MUST be removed from status when it is no longer active. List order has no semantic meaning. A status address matches an entry when its effective type matches, its value matches when requested, and its routability matches for a Cluster or prefixed request. An unset or empty request matches either an explicit empty value or Cluster. If every spec entry is satisfied and there are no retained active addresses, status.addresses contains the same number of entries as spec.addresses. If only some entries are satisfied, status.addresses contains only the successful entries, plus any retained active addresses.

The implicit-to-explicit transition described in Open Questions is excluded from these status rules until the behavior is defined for GA.

When spec.addresses is empty, implementations continue to populate status.addresses as they do today. Implementations claiming GatewayAddressRoutability MUST populate routability for every status address.

Full and Partially Accepted Address Entry Semantics

A new AddressesAssigned condition is added to Gateway status to surface address assignment outcomes independently of Programmed:

const (
	GatewayConditionAddressesAssigned GatewayConditionType = "AddressesAssigned"

	GatewayReasonAddressesAssigned          GatewayConditionReason = "AddressesAssigned"
	GatewayReasonAddressesPartiallyAssigned GatewayConditionReason = "PartiallyAssigned"
	GatewayReasonAddressesNotAssigned       GatewayConditionReason = "NotAssigned"
)

The condition MUST be omitted until the controller has reconciled address assignment. It is not added to the default Gateway conditions. After reconciliation, it applies both to explicit requests and to default address selection when spec.addresses is empty.

If all requested entries can be satisfied, or if default address selection has completed when spec.addresses is empty:

  • MUST set AddressesAssigned=True with reason Assigned

If some, but not all, entries can be satisfied, the implementation SHOULD program the Gateway using the addresses it can satisfy. In either case, it:

  • MUST set AddressesAssigned=False with reason PartiallyAssigned, with a message enumerating the unsatisfied entries.
  • MUST display all satisfied addresses in status.addresses; it MAY also display retained active addresses.

Vendors that opt to reject partially satisfied address entries MUST follow the same semantics as the “no entries can be satisfied” behavior below.

If no entries can be satisfied, the Gateway MUST NOT be programmed unless it is still using a retained active address. The implementation

  • MUST set Programmed=False with reason AddressNotAssigned unless it is still using a retained active address
  • MUST set AddressesAssigned=False with reason NotAssigned

Programmed otherwise retains its existing meaning: it reports whether the proxy is actually deployed and ready. A Gateway with all addresses assigned may still have Programmed=False for an unrelated reason.

Status Semantics

Each address in status.addresses from an implementation claiming GatewayAddressRoutability MUST have routability set, including an explicit empty value for the default behavior. An unset routability in status is understood by consumers as empty, preserving compatibility with implementations that do not claim the feature.

Addressing Backward Compatibility

For backward compatibility, an unset status.addresses[].routability is interpreted by consumers as empty. Implementations claiming GatewayAddressRoutability MUST report routability for every status address.

Default Semantics

  • An omitted or empty spec value does not constrain routability. An implementation claiming GatewayAddressRoutabilityCluster MUST report Cluster in status when it provides the Cluster guarantee; otherwise it MUST explicitly report an empty value.
  • An implementation MUST NOT report Cluster unless it claims GatewayAddressRoutabilityCluster and satisfies the Cluster requirements in this GEP.

Address Equivalence

When a Gateway supplies multiple addresses that share the same effective attributes – routability value, IP family (IPv4 or IPv6), and any future or implementation-specific per-address attributes – traffic to any of those addresses SHOULD produce equivalent results. Implementations SHOULD NOT specialize listener or routing behavior within such a set. This definition is intentionally extensible: as new per-address attributes are introduced, they narrow the equivalence class rather than conflicting with it.

Exceptions to Address Equivalence

  • Draining and rotating load balancers – all listeners may not drain at the same rate.
  • Making equivalence a MUST implies an admission policy which is out of scope for this proposal. If an operator wishes to deny clients access to a particular address on a particular listener, this should be allowed.
  • Per-address or per-client authn/authz decisions remain permitted.

Hostname Addresses

For addresses of type: Hostname, the routability value is expected to apply to any addresses the hostname resolves to. That is, a type: Hostname address with routability: Cluster carries the same reachability expectations as a type: IPAddress with routability: Cluster.

Cluster has portable semantics only for IPAddress. An implementation MAY support Cluster for a Hostname, but in that case it MUST guarantee that every address returned by ordinary in-cluster DNS resolution is in the cluster’s ServiceCIDR. Otherwise, it MUST leave the request unsatisfied. This behavior is implementation-specific and is not covered by portable conformance. NamedAddress and implementation-specific address types are likewise implementation-specific and have no portable Cluster guarantee.

Examples

Request a cluster-internal-only Gateway:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: internal-gw
spec:
  gatewayClassName: example
  addresses:
    - type: IPAddress
      routability: Cluster
  listeners:
    - name: http
      port: 80
      protocol: HTTP

Status showing a multi-address Gateway:

status:
  addresses:
    - type: IPAddress
      value: "203.0.113.10"
      routability: ""
    - type: IPAddress
      value: "10.96.0.42"
      routability: Cluster

Open Questions

Implicit-to-explicit address transitions

This GEP does not define whether the old address must be retained or removed, when the new address becomes active, how assignment conditions represent the transition, or whether traffic continuity is provided. Portable conformance does not test this transition. GA must define the transition contract, including status and condition semantics, handoff guarantees, and whether an explicit opt-in or migration mechanism is required.

Routability updates

Except for the implicit-to-explicit transition described above, routability is mutable at the API level. An implementation MAY accept or reject a routability update. When it rejects an update, it MUST treat the updated entry as unsatisfied and report the result through the assignment-condition semantics in this GEP. If the previous address remains active because it cannot be removed or replaced, the implementation MAY retain it in status until it is no longer active. The explanatory status message and any address-transition behavior are implementation-specific. Portable conformance does not test update behavior.

Per-address attributes

  • Some implementations may need per-address configuration beyond routability (e.g. load balancer class, traffic policy). If so, the principle should be that addresses sharing the same routability value and IP family MUST produce equivalent routing results, but MAY carry distinct implementation-specific metadata. Whether and how to expose such metadata (and how to avoid an open-ended map[string]string) is currently deferred to a follow-on proposal.

Conformance Details

Feature Names

GatewayAddressRoutability is an Extended feature. A GatewayClass that claims it through status.supportedFeatures MUST support the default behavior for omitted and empty IPAddress requests, support and report testing.x-k8s.io/sentinel, report routability on every status address, and implement the assignment-condition semantics in this GEP. Other implementation-specific routability values remain implementation-specific.

GatewayAddressRoutabilityCluster is an Extended feature. It MUST only be claimed with GatewayAddressRoutability. A GatewayClass that claims it MUST support Cluster IPAddress requests and the corresponding status and assignment semantics. Prefixed scopes and non-IP Cluster support remain implementation-specific.

Conformance test scenarios

Conformance tests for GatewayAddressRoutability will cover the following scenarios:

  • API validation accepts an empty string, Cluster, and prefixed values with a valid DNS subdomain prefix and non-empty path, including testing.x-k8s.io/sentinel. It rejects an unknown bare value, malformed prefixed values, invalid prefixes, and values whose prefix is k8s.io or ends in .k8s.io. The validation is tested on both spec and status addresses, including reserved-prefix examples such as k8s.io/value, example.k8s.io/value, and gateway.networking.k8s.io/value.
  • A GatewayClass claiming GatewayAddressRoutabilityCluster also claims GatewayAddressRoutability.
  • A GatewayClass claiming GatewayAddressRoutability accepts a testing.x-k8s.io/sentinel request and reports that exact routability value in status. This verifies support for routability reporting; it does not validate the assigned address or a routability guarantee.
  • A GatewayClass claiming GatewayAddressRoutabilityCluster reports a Cluster IPAddress request with routability: Cluster; the address is in the configured or discovered ServiceCIDR. In-cluster reachability is an implementation integration check, not a portable conformance assertion.
  • A GatewayClass claiming GatewayAddressRoutabilityCluster with one empty and one Cluster request reports exactly one matching status address for each, regardless of list order, and AddressesAssigned=True with reason Assigned. The empty request is reported as Cluster if it satisfies the Cluster requirements; otherwise it is explicitly reported as empty.
  • A Cluster IPAddress request with a static value outside the ServiceCIDR is unsatisfied. Combined with a satisfiable empty request, an implementation that permits partial assignment reports only the successful address and AddressesAssigned=False with reason PartiallyAssigned, with a message identifying the unsatisfied request. An implementation that rejects partial assignment follows the no-entries-can-be-satisfied behavior. On its own, the unsatisfied request reports Programmed=False with reason AddressNotAssigned and AddressesAssigned=False with reason NotAssigned. This scenario requires SupportGatewayStaticAddresses and GatewayAddressRoutabilityCluster.
  • A Cluster IPAddress request with a static value in the ServiceCIDR reports that exact value with routability: Cluster. This scenario requires SupportGatewayStaticAddresses and GatewayAddressRoutabilityCluster.
  • A Gateway with spec.addresses unset has AddressesAssigned=True with reason Assigned after default address selection. A claiming implementation reports Cluster for each resulting address that satisfies the Cluster requirements when it claims GatewayAddressRoutabilityCluster; otherwise it explicitly reports an empty value.
  • A claiming implementation with spec.addresses[].routability unset or explicitly empty reports Cluster when the resulting address satisfies the Cluster requirements and it claims GatewayAddressRoutabilityCluster; otherwise it explicitly reports an empty value.

The suite may discover ServiceCIDRs from the cluster or receive them through the serviceCIDRs conformance option. In-cluster reachability is not asserted by the portable suite because it does not provide a client workload in the target cluster. Tests for valid but unsupported prefixed values require configuration identifying a prefix that the implementation does not support and are not mandatory portable conformance tests. Address retention during rejected updates is excluded from portable conformance. Hostname, NamedAddress, and implementation-specific address types are excluded from portable conformance for this feature.

Alternatives Considered

  • Gateway-level spec.infrastructure.routability (GEP-1651). The predecessor design placed a single routability field on the Gateway rather than on each address. GEP-1651’s own “Alternatives” section anticipated per-address routability but recorded concerns about “complicating the Gateway’s purpose” by allowing multiple scopes. This GEP takes the position that multi-address, mixed-scope Gateways are a real and important use case (notably for egress, where a Gateway may need both a cluster-internal and a broader reachability scope), and that the per-address model is the cleaner way to express it. See Background for the full rationale.

  • Adding ClusterIP as a new AddressType. This conflates the reachability scope with the address format. A ClusterIP is an IPAddress with cluster scope, not a different type of address. Keeping type and routability orthogonal is cleaner and more extensible.

  • Defaulting omitted routability requests to External. Rejected: existing address-selection behavior is broader than external reachability, so that default could impose a breaking requirement. Omission and an empty value instead select the implementation default without a portable reachability guarantee.

References