GEP-713: Metaresources and Policy Attachment_metaresource_

• Issue: #713
• Status: Experimental

Note: This GEP is exempt from the Probationary Period rules of our GEP overview as it existed before those rules did, and so it has been explicitly grandfathered in.

TLDR

This GEP aims to standardize terminology and processes around using one Kubernetes object to modify the functions of one or more other objects.

This GEP defines some terms, firstly: Metaresource.

A Kubernetes object that that augments the behavior of an object in a standard way is called a Metaresource.

This document proposes controlling the creation of configuration in the underlying Gateway data plane using two types of Policy Attachment. A "Policy Attachment" is a specific type of metaresource that can affect specific settings across either one object (this is "Direct Policy Attachment"), or objects in a hierarchy (this is "Inherited Policy Attachment").

Individual policy APIs: - must be their own CRDs (e.g. TimeoutPolicy, RetryPolicy etc), - can be included in the Gateway API API group and installation or be defined by implementations - and must include a common TargetRef struct in their specification to identify how and where to apply that policy. - may include either a defaults section, an overrides section, or both. If these are included, the Policy is an Inherited Policy, and should use the inheritance rules defined in this document.

For Inherited Policies, this GEP also describes a set of expected behaviors for how settings can flow across a defined hierarchy.

Goals

• Establish a pattern for Policy resources which will be used for any policies included in the Gateway API spec
• Establish a pattern for Policy attachment, whether Direct or Inherited, which must be used for any implementation specific policies used with Gateway API resources
• Provide a way to distinguish between required and default values for all policy API implementations
• Enable policy attachment at all relevant scopes, including Gateways, Routes, Backends, along with how values should flow across a hierarchy if necessary
• Ensure the policy attachment specification is generic and forward thinking enough that it could be easily adapted to other grouping mechanisms like Namespaces in the future
• Provide a means of attachment that works for both ingress and mesh implementations of this API
• Provide a consistent specification that will ensure familiarity between both included and implementation-specific policies so they can both be interpreted the same way.

Out of scope

• Define all potential policies that may be attached to resources
• Design the full structure and configuration of policies

Background and concepts

When designing Gateway API, one of the things we’ve found is that we often need to be able change the behavior of objects without being able to make changes to the spec of those objects. Sometimes, this is because we can’t change the spec of the object to hold the information we need ( ReferenceGrant, from GEP-709, affecting Secrets and Services is an example, as is Direct Policy Attachment), and sometimes it’s because we want the behavior change to flow across multiple objects (this is what Inherited Policy Attachment is for).

To put this another way, sometimes we need ways to be able to affect how an object is interpreted in the API, without representing the description of those effects inside the spec of the object.

This document describes the ways we design objects to meet these two use cases, and why you might choose one or the other.

We use the term “metaresource” to describe the class of objects that only augment the behavior of another Kubernetes object, regardless of what they are targeting.

“Meta” here is used in its Greek sense of “more comprehensive” or “transcending”, and “resource” rather than “object” because “metaresource” is more pronounceable than “metaobject”. Additionally, a single word is better than a phrase like “wrapper object” or “wrapper resource” overall, although both of those terms are effectively synonymous with “metaresource”.

A "Policy Attachment" is a metaresource that affects the fields in existing objects (like Gateway or Routes), or influences the configuration that's generated in an underlying data plane.

"Direct Policy Attachment" is when a Policy object references a single object only, and only modifies the fields of or the configuration associated with that object.

"Inherited Policy Attachment" is when a Policy object references a single object and any child objects of that object (according to some defined hierarchy), and modifies fields of the child objects, or configuration associated with the child objects.

In either case, a Policy may either affect an object by controlling the value of one of the existing fields in the spec of an object, or it may add additional fields that are not in the spec of the object.

Direct Policy Attachment

A Direct Policy Attachment is tightly bound to one instance of a particular Kind within a single namespace (or to an instance of a single Kind at cluster scope), and only modifies the behavior of the object that matches its binding.

As an example, one use case that Gateway API currently does not support is how to configure details of the TLS required to connect to a backend (in other words, if the process running inside the backend workload expects TLS, not that some automated infrastructure layer is provisioning TLS as in the Mesh case).

A hypothetical TLSConnectionPolicy that targets a Service could be used for this, using the functionality of the Service as describing a set of endpoints. (It should also be noted this is not the only way to solve this problem, just an example to illustrate Direct Policy Attachment.)

The TLSConnectionPolicy would look something like this:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSConnectionPolicy
metadata:
  name: tlsport8443
  namespace: foo
spec:
  targetRef: # This struct is defined as part of Gateway API
    group: "" # Empty string means core - this is a standard convention
    kind: Service
    name: fooService
  tls:
    certificateAuthorityRefs:
      - name: CAcert
    port: 8443

All this does is tell an implementation, that for connecting to port 8443 on the Service fooService, it should assume that the connection is TLS, and expect the service's certificate to be validated by the chain in the CAcert Secret.

Importantly, this would apply to every usage of that Service across any HTTPRoutes in that namespace, which could be useful for a Service that is reused in a lot of HTTPRoutes.

With these two examples in mind, here are some guidelines for when to consider using Direct Policy Attachment:

• The number or scope of objects to be modified is limited or singular. Direct Policy Attachments must target one specific object.
• The modifications to be made to the objects don’t have any transitive information - that is, the modifications only affect the single object that the targeted metaresource is bound to, and don’t have ramifications that flow beyond that object.
• In terms of status, it should be reasonably easy for a user to understand that everything is working - basically, as long as the targeted object exists, and the modifications are valid, the metaresource is valid, and this should be straightforward to communicate in one or two Conditions. Note that at the time of writing, this is not completed.
• Direct Policy Attachment should only be used to target objects in the same namespace as the Policy object. Allowing cross-namespace references brings in significant security concerns, and/or difficulties about merging cross-namespace policy objects. Notably, Mesh use cases may need to do something like this for consumer policies, but in general, Policy objects that modify the behavior of things outside their own namespace should be avoided unless it uses a handshake of some sort, where the things outside the namespace can opt–out of the behavior. (Notably, this is the design that we used for ReferenceGrant).

Inherited Policy Attachment: It's all about the defaults and overrides

Because a Inherited Policy is a metaresource, it targets some other resource and augments its behavior.

But why have this distinct from other types of metaresource? Because Inherited Policy resources are designed to have a way for settings to flow down a hierarchy.

Defaults set the default value for something, and can be overridden by the “lower” objects (like a connection timeout default policy on a Gateway being overridable inside a HTTPRoute), and Overrides cannot be overridden by “lower” objects (like setting a maximum client timeout to some non-infinite value at the Gateway level to stop HTTPRoute owners from leaking connections over time).

Here are some guidelines for when to consider using a Inherited Policy object:

• The settings or configuration are bound to one containing object, but affect other objects attached to that one (for example, affecting HTTPRoutes attached to a single Gateway, or all HTTPRoutes in a GatewayClass).
• The settings need to able to be defaulted, but can be overridden on a per-object basis.
• The settings must be enforced by one persona, and not modifiable or removable by a lesser-privileged persona. (The owner of a GatewayClass may want to restrict something about all Gateways in a GatewayClass, regardless of who owns the Gateway, or a Gateway owner may want to enforce some setting across all attached HTTPRoutes).
• In terms of status, a good accounting for how to record that the Policy is attached is easy, but recording what resources the Policy is being applied to is not, and needs to be carefully designed to avoid fanout apiserver load. (This is not built at all in the current design either).

When multiple Inherited Policies are used, they can interact in various ways, which are governed by the following rules, which will be expanded on later in in this document.

• If a Policy does not affect an object's fields directly, then the resultant Policy should be the set of all distinct fields inside the relevant Policy objects, as set out by the rules below.
• For Policies that affect an object's existing fields, multiple instances of the same Policy Kind affecting an object's fields will be evaluated as though only a single Policy "wins" the right to affect each field. This operation is performed on a per-distinct-field basis.
• Settings in overrides stanzas will win over the same setting in a defaults stanza.
• overrides settings operate in a "less specific beats more specific" fashion - Policies attached higher up the hierarchy will beat the same type of Policy attached further down the hierarchy.
• defaults settings operate in a "more specific beats less specific" fashion - Policies attached lower down the hierarchy will beat the same type of Policy attached further up the hierarchy.
• For defaults, the most specific value is the one inside the object that the Policy applies to; that is, if a Policy specifies a default, and an object specifies a value, the object's value will win.
• Policies interact with the fields they are controlling in a "replace value" fashion.
• For fields where the value is a scalar, (like a string or a number) should have their value replaced by the value in the Policy if it wins. Notably, this means that a default will only ever replace an empty or unset value in an object.
• For fields where the value is an object, the Policy should include the fields in the object in its definition, so that the replacement can be on simple fields rather than complex ones.
• For fields where the final value is non-scalar, but is not an object with fields of its own, the value should be entirely replaced, not merged. This means that lists of strings or lists of ints specified in a Policy will overwrite the empty list (in the case of a default) or any specified list (in the case of an override). The same applies to map[string]string fields. An example here would be a field that stores a map of annotations - specifying a Policy that overrides annotations will mean that a final object specifying those annotations will have its value entirely replaced by an override setting.
• In the case that two Policies of the same type specify different fields, then all of the specified fields should take effect on the affected object.

Examples to further illustrate these rules are given below.

API

This approach is building on concepts from all of the alternatives discussed below. This is very similar to the (now removed) BackendPolicy resource in the API, but also borrows some concepts from the ServicePolicy proposal.

Policy Attachment for Ingress

Attaching a Directly Attached Policy to Gateway resources for ingress use cases is relatively straightforward. A policy can reference the resource it wants to apply to.

Access is granted with RBAC - anyone that has access to create a RetryPolicy in a given namespace can attach it to any resource within that namespace.

An Inherited Policy can attach to a parent resource, and then each policy applies to the referenced resource and everything below it in terms of hierarchy. Although this example is likely more complex than many real world use cases, it helps demonstrate how policy attachment can work across namespaces.

Policy Attachment for Mesh

Although there is a great deal of overlap between ingress and mesh use cases, mesh enables more complex policy attachment scenarios. For example, you may want to apply policy to requests from a specific namespace to a backend in another namespace.

Policy attachment can be quite simple with mesh. Policy can be applied to any resource in any namespace but it can only apply to requests from the same namespace if the target is in a different namespace.

At the other extreme, policy can be used to apply to requests from a specific workload to a backend in another namespace. A route can be used to intercept these requests and split them between different backends (foo-a and foo-b in this case).

Policy TargetRef API

Each Policy resource MUST include a single targetRef field. It must not target more than one resource at a time, but it can be used to target larger resources such as Gateways or Namespaces that may apply to multiple child resources.

As with most APIs, there are countless ways we could choose to expand this in the future. This includes supporting multiple targetRefs and/or label selectors. Although this would enable compelling functionality, it would increase the complexity of an already complex API and potentially result in more conflicts between policies. Although we may choose to expand the targeting capabilities in the future, at this point it is strongly preferred to start with a simpler pattern that still leaves room for future expansion.

The targetRef field MUST have the following structure:

// PolicyTargetReference identifies an API object to apply policy to.
type PolicyTargetReference struct {
    // Group is the group of the target resource.
    //
    // +kubebuilder:validation:MinLength=1
    // +kubebuilder:validation:MaxLength=253
    Group string `json:"group"`

    // Kind is kind of the target resource.
    //
    // +kubebuilder:validation:MinLength=1
    // +kubebuilder:validation:MaxLength=253
    Kind string `json:"kind"`

    // Name is the name of the target resource.
    //
    // +kubebuilder:validation:MinLength=1
    // +kubebuilder:validation:MaxLength=253
    Name string `json:"name"`

    // Namespace is the namespace of the referent. When unspecified, the local
    // namespace is inferred. Even when policy targets a resource in a different
    // namespace, it may only apply to traffic originating from the same
    // namespace as the policy.
    //
    // +kubebuilder:validation:MinLength=1
    // +kubebuilder:validation:MaxLength=253
    // +optional
    Namespace string `json:"namespace,omitempty"`
}

Sample Policy API

The following structure can be used as a starting point for any Policy resource using this API pattern. Note that the PolicyTargetReference struct defined above will be distributed as part of the Gateway API.

// ACMEServicePolicy provides a way to apply Service policy configuration with
// the ACME implementation of the Gateway API.
type ACMEServicePolicy struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    // Spec defines the desired state of ACMEServicePolicy.
    Spec ACMEServicePolicySpec `json:"spec"`

    // Status defines the current state of ACMEServicePolicy.
    Status ACMEServicePolicyStatus `json:"status,omitempty"`
}

// ACMEServicePolicySpec defines the desired state of ACMEServicePolicy.
type ACMEServicePolicySpec struct {
    // TargetRef identifies an API object to apply policy to.
    TargetRef gatewayv1a2.PolicyTargetReference `json:"targetRef"`

    // Override defines policy configuration that should override policy
    // configuration attached below the targeted resource in the hierarchy.
    // +optional
    Override *ACMEPolicyConfig `json:"override,omitempty"`

    // Default defines default policy configuration for the targeted resource.
    // +optional
    Default *ACMEPolicyConfig `json:"default,omitempty"`
}

// ACMEPolicyConfig contains ACME policy configuration.
type ACMEPolicyConfig struct {
    // Add configurable policy here
}

// ACMEServicePolicyStatus defines the observed state of ACMEServicePolicy.
type ACMEServicePolicyStatus struct {
    // Conditions describe the current conditions of the ACMEServicePolicy.
    //
    // +optional
    // +listType=map
    // +listMapKey=type
    // +kubebuilder:validation:MaxItems=8
    Conditions []metav1.Condition `json:"conditions,omitempty"`
}

Hierarchy

Each policy MAY include default or override values. Default values are given precedence from the bottom up, while override values are top down. That means that a default attached to a Backend will have the highest precedence among default values while an override value attached to a GatewayClass will have the highest precedence overall.

To illustrate this, consider 3 resources with the following hierarchy: A > B > C. When attaching the concept of defaults and overrides to that, the hierarchy would be expanded to this:

A override > B override > C override > C default > B default > A default.

Note that the hierarchy is reversed for defaults. The rationale here is that overrides usually need to be enforced top down while defaults should apply to the lowest resource first. For example, if an admin needs to attach required policy, they can attach it as an override to a Gateway. That would have precedence over Routes and Services below it. On the other hand, an app owner may want to set a default timeout for their Service. That would have precedence over defaults attached at higher levels such as Route or Gateway.

If using defaults and overrides, each policy resource MUST include 2 structs within the spec. One with override values and the other with default values.

In the following example, the policy attached to the Gateway requires cdn to be enabled and provides some default configuration for that. The policy attached to the Route changes the value for one of those fields (includeQueryString).

kind: CDNCachingPolicy # Example of implementation specific policy name
spec:
  override:
    cdn:
      enabled: true
  default:
    cdn:
      cachePolicy:
        includeHost: true
        includeProtocol: true
        includeQueryString: true
  targetRef:
    kind: Gateway
    name: example
---
kind: CDNCachingPolicy
spec:
  default:
    cdn:
      cachePolicy:
        includeQueryString: false
  targetRef:
    type: direct
    kind: HTTPRoute
    name: example

In this final example, we can see how the override attached to the Gateway has precedence over the default drainTimeout value attached to the Route. At the same time, we can see that the default connectionTimeout attached to the Route has precedence over the default attached to the Gateway.

Also note how the different resources interact - fields that are not common across objects may both end up affecting the final object.

Supported Resources

It is important to note that not every implementation will be able to support policy attachment to each resource described in the hierarchy above. When that is the case, implementations MUST clearly document which resources a policy may be attached to.

Attaching Policy to GatewayClass

GatewayClass may be the trickiest resource to attach policy to. Policy attachment relies on the policy being defined within the same scope as the target. This ensures that only users with write access to a policy resource in a given scope will be able to modify policy at that level. Since GatewayClass is a cluster scoped resource, this means that any policy attached to it must also be cluster scoped.

GatewayClass parameters provide an alternative to policy attachment that may be easier for some implementations to support. These parameters can similarly be used to set defaults and requirements for an entire GatewayClass.

Targeting External Services

In some cases (likely limited to mesh) we may want to apply policies to requests to external services. To accomplish this, implementations can choose to support a reference to a virtual resource type:

apiVersion: networking.acme.io/v1alpha1
kind: RetryPolicy
metadata:
  name: foo
spec:
  default:
    maxRetries: 5
  targetRef:
    group: networking.acme.io
    kind: ExternalService
    name: foo.com

Merging into existing spec fields

It's possible (even likely) that configuration in a Policy may need to be merged into an existing object's fields somehow, particularly for Inherited policies.

When merging into an existing fields inside an object, Policy objects should merge values at a scalar level, not at a struct or object level.

For example, in the CDNCachingPolicy example above, the cdn struct contains a cachePolicy struct that contains fields. If an implementation was merging this configuration into an existing object that contained the same fields, it should merge the fields at a scalar level, with the includeHost, includeProtocol, and includeQueryString values being defaulted if they were not specified in the object being controlled. Similarly, for overrides, the values of the innermost scalar fields should overwrite the scalar fields in the affected object.

Implementations should not copy any structs from the Policy object directly into the affected object, any fields that are overridden should be overridden on a per-field basis.

In the case that the field in the Policy affects a struct that is a member of a list, each existing item in the list in the affected object should have each of its fields compared to the corresponding fields in the Policy.

For non-scalar field values, like a list of strings, or a map[string]string value, the entire value must be overwritten by the value from the Policy. No merging should take place. This mainly applies to overrides, since for defaults, there should be no value present in a field on the final object.

This table shows how this works for various types:

Type	Object config	Override Policy config	Result
string	key: "foo"	key: "bar"	key: "bar"
list	key: ["a","b"]	key: ["c","d"]	key: ["c","d"]
map[string]string	key: {"foo": "a", "bar": "b"}	key: {"foo": "c", "bar": "d"}	key: {"foo": "c", "bar": "d"}

Conflict Resolution

It is possible for multiple policies to target the same object and the same fields inside that object. If multiple policy resources target the same resource and have an identical field specified with different values, precedence MUST be determined in order of the following criteria, continuing on ties:

• Direct Policies override Inherited Policies. If preventing settings from being overwritten is important, implementations should only use Inherited Policies, and the override stanza that implies. Note also that it's not intended that Direct and Inherited Policies should overlap, so this should only come up in exceptional circumstances.
• Inside Inherited Policies, the same setting in overrides beats the one in defaults.
• The oldest Policy based on creation timestamp. For example, a Policy with a creation timestamp of "2021-07-15 01:02:03" is given precedence over a Policy with a creation timestamp of "2021-07-15 01:02:04".
• The Policy appearing first in alphabetical order by {namespace}/{name}. For example, foo/bar is given precedence over foo/baz.

For a better user experience, a validating webhook can be implemented to prevent these kinds of conflicts all together.

Kubectl Plugin

To help improve UX and standardization, a kubectl plugin will be developed that will be capable of describing the computed sum of policy that applies to a given resource, including policies applied to parent resources.

Each Policy CRD that wants to be supported by this plugin will need to follow the API structure defined above and add a gateway.networking.k8s.io/policy: true label to the CRD.

Status

In the current iteration of this GEP, metaresources and Policy objects don't have any standard way to record what they're attaching to, or applying settings to in the case of Policy Attachment. There are some recommended Condition types defined below, but further work on the status design is required to ensure that some problems are resolved:

• When multiple controllers are implementing the same Route and recognize a policy, it must be possible to determine which controller was responsible for adding that policy reference to status. Adding Conditions to status on the Policy instead can be helpful here, but we're still lacking a way for the Route or Gateway owner to find all the Policies that are influencing their object.
• For this to be somewhat scalable, we must limit the number of status updates that can result from a metaresource update.
• Since we only control some of the resources a policy might be attached to, adding policies to status would only be possible on the policy objects themselves or on Gateway API resources, not Services or other kinds of backends.

Previous experience in the Kubernetes API has made it clear that having a single object that can cause status updates to occur across many other objects can have a big performance impact, so the status design must be very carefully done to avoid these kind of fanout problems.

However, the whole purpose of having a standardized Policy API structure and patterns is intended to make this problem solvable both for human users and with tooling.

This is currently a very open question. A discussion is ongoing at #1531, and this GEP will be updated with any outcomes.

Conditions

Controllers using the Gateway API policy attachment model SHOULD populate the following condition and reasons on policy resources to provide a consistent experience across implementations.

// PolicyConditionType is a type of condition for a policy.
type PolicyConditionType string

// PolicyConditionReason is a reason for a policy condition.
type PolicyConditionReason string

const (
  // PolicyConditionAccepted indicates whether the policy has been accepted or rejected
  // by a targeted resource, and why.
  //
  // Possible reasons for this condition to be True are:
  //
  // * "Accepted"
  //
  // Possible reasons for this condition to be False are:
  //
  // * "Conflicted"
  // * "Invalid"
  // * "TargetNotFound"
  //
  PolicyConditionAccepted PolicyConditionType = "Accepted"

  // PolicyReasonAccepted is used with the "Accepted" condition when the policy has been
  // accepted by the targeted resource.
  PolicyReasonAccepted PolicyConditionReason = "Accepted"

  // PolicyReasonConflicted is used with the "Accepted" condition when the policy has not
  // been accepted by a targeted resource because there is another policy that targets the same
  // resource and a merge is not possible.
  PolicyReasonConflicted PolicyConditionReason = "Conflicted"

  // PolicyReasonInvalid is used with the "Accepted" condition when the policy is syntactically
  // or semantically invalid.
  PolicyReasonInvalid PolicyConditionReason = "Invalid"

  // PolicyReasonTargetNotFound is used with the "Accepted" condition when the policy is attached to
  // an invalid target resource
  PolicyReasonTargetNotFound PolicyConditionReason = "TargetNotFound"
)

Interaction with Custom Filters and other extension points

There are multiple methods of custom extension in the Gateway API. Policy attachment and custom Route filters are two of these. Policy attachment is designed to provide arbitrary configuration fields that decorate Gateway API resources. Route filters provide custom request/response filters embedded inside Route resources. Both are extension methods for fields that cannot easily be standardized as core or extended fields of the Gateway API. The following guidance should be considered when introducing a custom field into any Gateway controller implementation:

1. For any given field that a Gateway controller implementation needs, the possibility of using core or extended should always be considered before using custom policy resources. This is encouraged to promote standardization and, over time, to absorb capabilities into the API as first class fields, which offer a more streamlined UX than custom policy attachment.

2. Although it's possible that arbitrary fields could be supported by custom policy, custom route filters, and core/extended fields concurrently, it is recommended that implementations only use multiple mechanisms for representing the same fields when those fields really need the defaulting and/or overriding behavior that Policy Attachment provides. For example, a custom filter that allowed the configuration of Authentication inside a HTTPRoute object might also have an associated Policy resource that allowed the filter's settings to be defaulted or overridden. It should be noted that doing this in the absence of a solution to the status problem is likely to be very difficult to troubleshoot.

Conformance Level

This policy attachment pattern is associated with an "EXTENDED" conformance level. The implementations that support this policy attachment model will have the same behavior and semantics, although they may not be able to support attachment of all types of policy at all potential attachment points.

Apply Policies to Sections of a Resource (Future Extension)

Although initially out of scope, it would be helpful to be able to target specific matches within nested objects. For example, it may be useful to attach policies to a specific Gateway listener or Route rule. This section explores what that could look like.

Each Route rule or Gateway listener should be expanded with an optional name field. The target ref would be expanded with an optional sectionName field that could be used to refer to that specific section of the resource. It would refer to the following concepts on these resources:

• Gateway.Listeners.Name
• xRoute.Rules.Name
• Service.Ports.Name

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: http-app-1
  labels:
    app: foo
spec:
  hostnames:
  - "foo.com"
  rules:
  - name: bar
    matches:
    - path:
        type: Prefix
        value: /bar
    forwardTo:
    - serviceName: my-service1
      port: 8080
---
apiVersion: networking.acme.io/v1alpha2
kind: RetryPolicy
metadata:
  name: foo
spec:
  maxRetries: 5
  targetRef:
    name: foo
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    sectionName: bar

This would require adding a SectionName field to the PolicyTargetReference:

type PolicyTargetReference struct {
    // SectionName is the name of a section within the target resource. When
    // unspecified, this targets the entire resource. In the following
    // resources, SectionName is interpreted as the following:
    // * Gateway: Listener Name
    // * Route: Rule Name
    // * Service: Port Name
    //
    // +kubebuilder:validation:MinLength=1
    // +kubebuilder:validation:MaxLength=253
    // +optional
    SectionName string `json:"sectionName,omitempty"`
    // ...
}

This would also require adding a Name field to Gateway listeners and Route rules:

type Listener struct {
    // Name is the name of the Listener. If more than one Listener is present
    // each Listener MUST specify a name. The names of Listeners MUST be unique
    // within a Gateway.
    //
    // Support: Core
    //
    // +kubebuilder:validation:MinLength=1
    // +kubebuilder:validation:MaxLength=253
    // +optional
    Name string `json:"name,omitempty"`
    // ...
}

type RouteRule struct {
    // Name is the name of the Route rule. If more than one Route Rule is
    // present, each Rule MUST specify a name. The names of Rules MUST be unique
    // within a Route.
    //
    // Support: Core
    //
    // +kubebuilder:validation:MinLength=1
    // +kubebuilder:validation:MaxLength=253
    // +optional
    Name string `json:"name,omitempty"`
    // ...
}

Advantages

• Incredibly flexible approach that should work well for both ingress and mesh
• Conceptually similar to existing ServicePolicy proposal and BackendPolicy pattern
• Easy to attach policy to resources we don’t control (Service, ServiceImport, etc)
• Minimal API changes required
• Simplifies packaging an application for deployment as policy references do not need to be part of the templating

Disadvantages

• May be difficult to understand which policies apply to a request

Examples

This section provides some examples of various types of Policy objects, and how merging, defaults, overrides, and other interactions work.

Direct Policy Attachment

The following Policy sets the minimum TLS version required on a Gateway Listener:

apiVersion: networking.example.io/v1alpha1
kind: TLSMinimumVersionPolicy
metadata:
  name: minimum12
  namespace: appns
spec:
  minimumTLSVersion: 1.2
  targetRef:
    name: internet
    group: gateway.networking.k8s.io
    kind: Gateway

Note that because there is no version controlling the minimum TLS version in the Gateway spec, this is an example of a non-field Policy.

Inherited Policy Attachment

It also could be useful to be able to default the minimumTLSVersion setting across multiple Gateways.

This version of the above Policy allows this:

apiVersion: networking.example.io/v1alpha1
kind: TLSMinimumVersionPolicy
metadata:
  name: minimum12
  namespace: appns
spec:
  defaults:
    minimumTLSVersion: 1.2
  targetRef:
    name: appns
    group: ""
    kind: namespace

This Inherited Policy is using the implicit hierarchy that all resources belong to a namespace, so attaching a Policy to a namespace means affecting all possible resources in a namespace. Multiple hierarchies are possible, even within Gateway API, for example Gateway -> Route, Gateway -> Route -> Backend, Gateway -> Route -> Service. GAMMA Policies could conceivably use a hierarchy of Service -> Route as well.

Note that this will not be very discoverable for Gateway owners in the absence of a solution to the Policy status problem. This is being worked on and this GEP will be updated once we have a design.

Conceivably, a security or admin team may want to force Gateways to have at least a minimum TLS version of 1.2 - that would be a job for overrides, like so:

apiVersion: networking.example.io/v1alpha1
kind: TLSMinimumVersionPolicy
metadata:
  name: minimum12
  namespace: appns
spec:
  overrides:
    minimumTLSVersion: 1.2
  targetRef:
    name: appns
    group: ""
    kind: namespace

This will make it so that all Gateways in the default namespace must use a minimum TLS version of 1.2, and this cannot be changed by Gateway owners. Only the Policy owner can change this Policy.

Handling non-scalar values

In this example, we will assume that at some future point, HTTPRoute has grown fields to configure retries, including a field called retryOn that reflects the HTTP status codes that should be retried. The value of this field is a list of strings, being the HTTP codes that must be retried. The retryOn field has no defaults in the field definitions (which is probably a bad design, but we need to show this interaction somehow!)

We also assume that a Inherited RetryOnPolicy exists that allows both defaulting and overriding of the retryOn field.

A full RetryOnPolicy to default the field to the codes 501, 502, and 503 would look like this:

apiVersion: networking.example.io/v1alpha1
kind: RetryOnPolicy
metadata:
  name: retryon5xx
  namespace: appns
spec:
  defaults:
    retryOn:
      - "501"
      - "502"
      - "503"
  targetRef:
    kind: Gateway
    group: gateway.networking.k8s.io
    name: we-love-retries

This means that, for HTTPRoutes that do NOT explicitly set this field to something else, (in other words, they contain an empty list), then the field will be set to a list containing 501, 502, and 503. (Notably, because of Go zero values, this would also occur if the user explicitly set the value to the empty list.)

However, if a HTTPRoute owner sets any value other than the empty list, then that value will remain, and the Policy will have no effect. These values are not merged.

If the Policy used overrides instead:

apiVersion: networking.example.io/v1alpha1
kind: RetryOnPolicy
metadata:
  name: retryon5xx
  namespace: appns
spec:
  overrides:
    retryOn:
      - "501"
      - "502"
      - "503"
  targetRef:
    kind: Gateway
    group: gateway.networking.k8s.io
    name: you-must-retry

Then no matter what the value is in the HTTPRoute, it will be set to 501, 502, 503 by the Policy override.

Interactions between defaults, overrides, and field values

All HTTPRoutes that attach to the YouMustRetry Gateway will have any value overwritten by this policy. The empty list, or any number of values, will all be replaced with 501, 502, and 503.

Now, let's also assume that we use the Namespace -> Gateway hierarchy on top of the Gateway -> HTTPRoute hierarchy, and allow attaching a RetryOnPolicy to a namespace. The expectation here is that this will affect all Gateways in a namespace and all HTTPRoutes that attach to those Gateways. (Note that the HTTPRoutes themselves may not necessarily be in the same namespace though.)

If we apply the default policy from earlier to the namespace:

apiVersion: networking.example.io/v1alpha1
kind: RetryOnPolicy
metadata:
  name: retryon5xx
  namespace: appns
spec:
  defaults:
    retryOn:
      - "501"
      - "502"
      - "503"
  targetRef:
    kind: Namespace
    group: ""
    name: appns

Then this will have the same effect as applying that Policy to every Gateway in the default namespace - namely that every HTTPRoute that attaches to every Gateway will have its retryOn field set to 501, 502, 503, if there is no other setting in the HTTPRoute itself.

With two layers in the hierarchy, we have a more complicated set of interactions possible.

Let's look at some tables for a particular HTTPRoute, assuming that it does not configure the retryOn field, for various types of Policy at different levels.

Overrides interacting with defaults for RetryOnPolicy, empty list in HTTPRoute

	None	Namespace override	Gateway override	HTTPRoute override
No default	Empty list	Namespace override	Gateway override Policy	HTTPRoute override
Namespace default	Namespace default	Namespace override	Gateway override	HTTPRoute override
Gateway default	Gateway default	Namespace override	Gateway override	HTTPRoute override
HTTPRoute default	HTTPRoute default	Namespace override	Gateway override	HTTPRoute override

Overrides interacting with other overrides for RetryOnPolicy, empty list in HTTPRoute

	No override	Namespace override A	Gateway override A	HTTPRoute override A
No override	Empty list	Namespace override	Gateway override	HTTPRoute override
Namespace override B	Namespace override B	Namespace override first created wins otherwise first alphabetically	Namespace override B	Namespace override B
Gateway override B	Gateway override B	Namespace override A	Gateway override first created wins otherwise first alphabetically	Gateway override B
HTTPRoute override B	HTTPRoute override B	Namespace override A	Gateway override A	HTTPRoute override first created wins otherwise first alphabetically

Defaults interacting with other defaults for RetryOnPolicy, empty list in HTTPRoute

	No default	Namespace default A	Gateway default A	HTTPRoute default A
No default	Empty list	Namespace default	Gateway default	HTTPRoute default A
Namespace default B	Namespace default B	Namespace default first created wins otherwise first alphabetically	Gateway default A	HTTPRoute default A
Gateway default B	Gateway default B	Gateway default B	Gateway default first created wins otherwise first alphabetically	HTTPRoute default A
HTTPRoute default B	HTTPRoute default B	HTTPRoute default B	HTTPRoute default B	HTTPRoute default first created wins otherwise first alphabetically

Now, if the HTTPRoute does specify a RetryPolicy, it's a bit easier, because we can basically disregard all defaults:

Overrides interacting with defaults for RetryOnPolicy, value in HTTPRoute

	None	Namespace override	Gateway override	HTTPRoute override
No default	Value in HTTPRoute	Namespace override	Gateway override	HTTPRoute override
Namespace default	Value in HTTPRoute	Namespace override	Gateway override	HTTPRoute override
Gateway default	Value in HTTPRoute	Namespace override	Gateway override	HTTPRoute override
HTTPRoute default	Value in HTTPRoute	Namespace override	Gateway override	HTTPRoute override

Overrides interacting with other overrides for RetryOnPolicy, value in HTTPRoute

	No override	Namespace override A	Gateway override A	HTTPRoute override A
No override	Value in HTTPRoute	Namespace override A	Gateway override A	HTTPRoute override A
Namespace override B	Namespace override B	Namespace override first created wins otherwise first alphabetically	Namespace override B	Namespace override B
Gateway override B	Gateway override B	Namespace override A	Gateway override first created wins otherwise first alphabetically	Gateway override B
HTTPRoute override B	HTTPRoute override B	Namespace override A	Gateway override A	HTTPRoute override first created wins otherwise first alphabetically

Defaults interacting with other defaults for RetryOnPolicy, value in HTTPRoute

	No default	Namespace default A	Gateway default A	HTTPRoute default A
No default	Value in HTTPRoute	Value in HTTPRoute	Value in HTTPRoute	Value in HTTPRoute
Namespace default B	Value in HTTPRoute	Value in HTTPRoute	Value in HTTPRoute	Value in HTTPRoute
Gateway default B	Value in HTTPRoute	Value in HTTPRoute	Value in HTTPRoute	Value in HTTPRoute
HTTPRoute default B	Value in HTTPRoute	Value in HTTPRoute	Value in HTTPRoute	Value in HTTPRoute

Removing BackendPolicy

BackendPolicy represented the initial attempt to cover policy attachment for Gateway API. Although this proposal ended up with a similar structure to BackendPolicy, it is not clear that we ever found sufficient value or use cases for BackendPolicy. Given that this proposal provides more powerful ways to attach policy, BackendPolicy was removed.

Alternatives

1. ServiceBinding for attaching Policies and Routes for Mesh

A new ServiceBinding resource has been proposed for mesh use cases. This would provide a way to attach policies, including Routes to a Service.

Most notably, these provide a way to attach different policies to requests coming from namespaces or specific Gateways. In the example below, a ServiceBinding in the consumer namespace would be applied to the selected Gateway and affect all requests from that Gateway to the foo Service. Beyond policy attachment, this would also support attaching Routes as policies, in this case the attached HTTPRoute would split requests between the foo-a and foo-b Service instead of the foo Service.

This approach can be used to attach a default set of policies to all requests coming from a namespace. The example below shows a ServiceBinding defined in the producer namespace that would apply to all requests from within the same namespace or from other namespaces that did not have their own ServiceBindings defined.

Advantages

• Works well for mesh and any use cases where requests don’t always transit through Gateways and Routes.
• Allows policies to apply to an entire namespace.
• Provides very clear attachment of polices, routes, and more to a specific Service.
• Works well for ‘shrink-wrap application developers’ - the packaged app does not need to know about hostnames or policies or have extensive templates.
• Works well for ‘dynamic’ / programmatic creation of workloads ( Pods,etc - see CertManager)
• It is easy to understand what policy applies to a workload - by listing the bindings in the namespace.

Disadvantages

• Unclear how this would work with an ingress model. If Gateways, Routes, and Backends are all in different namespaces, and each of those namespaces has different ServiceBindings applying different sets of policies, it’s difficult to understand which policy would be applied.
• Unclear if/how this would interact with existing the ingress focused policy proposal described below. If both coexisted, would it be possible for a user to understand which policies were being applied to their requests?
• Route status could get confusing when Routes were referenced as a policy by ServiceBinding
• Introduces a new mesh specific resource.

2. Attaching Policies for Ingress

An earlier proposal for policy attachment in the Gateway API suggested adding policy references to each Resource. This works very naturally for Ingress use cases where all requests follow a path through Gateways, Routes, and Backends. Adding policy attachment at each level enables different roles to define defaults and allow overrides at different levels.

Advantages

• Consistent policy attachment at each level
• Clear which policies apply to each component
• Naturally translates to hierarchical Ingress model with ability to delegate policy decisions to different roles

Disadvantages

• Policy overrides could become complicated
• At least initially, policy attachment on Service would have to rely on Service annotations or references from policy to Service(s)
• No way to attach policy to other resources such as namespace or ServiceImport
• May be difficult to modify Routes and Services if other components/roles are managing them (eg Knative)

3. Shared Policy Resource

This is really just a slight variation or extension of the main proposal in this GEP. We would introduce a shared policy resource. This resource would follow the guidelines described above, including the targetRef as defined as well as default and override fields. Instead of carefully crafted CRD schemas for each of the default and override fields, we would use more generic map[string]string values. This would allow similar flexibility to annotations while still enabling the default and override concepts that are key to this proposal.

Unfortunately this would be difficult to validate and would come with many of the downsides of annotations. A validating webhook would be required for any validation which could result in just as much or more work to maintain than CRDs. At this point we believe that the best experience will be from implementations providing their own policy CRDs that follow the patterns described in this GEP. We may want to explore tooling or guidance to simplify the creation of these policy CRDs to help simplify implementation and extension of this API.

References

Issues * Extensible Service Policy and Configuration

Docs * Policy Attachment and Binding