Add support for AI APIs

[npolshakova]

Description

Adds support for AI APIs: #10494

API changes

Introduces AI APIs in kgateway. This is based on the Gloo AI APIs.

The HTTPRoute selects an Upstream via a backendRef:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: route-to-upstream
  namespace: gwtest
spec:
  parentRefs:
    - name: http-gw-for-test
  hostnames:
    - "test"
  rules:
    - matches:
        - path:
            type: Exact
            value: /v1/chat/completions
    - backendRefs:
        - name: openai
          kind: Upstream
          group: gateway.kgateway.dev

Where an Upstream specifies a provider and auth type:

apiVersion: gateway.kgateway.dev/v1alpha1
kind: Upstream
metadata:
  labels:
    app: kgateway
  name: anthropic
  namespace: gwtest
spec:
  ai:
    llm:
      provider:
        anthropic:
          authToken:
            kind: "Passthrough"

To select a RoutePolicy you can use extensionRef on the AI Upstream route:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: route-to-upstream
  namespace: gwtest
spec:
  parentRefs:
    - name: http-gw-for-test
  hostnames:
    - "test"
  rules:
    - matches:
        - path:
            type: Exact
            value: /v1/chat/completions
    - backendRefs:
        - name: openai
          kind: Upstream
          group: gateway.kgateway.dev
      filters:
      - type: ExtensionRef
        extensionRef:
           group: gateway.kgateway.dev/v1alpha1
           kind: RoutePolicy
           name: open-ai-opt
---
apiVersion: gateway.kgateway.dev/v1alpha1
kind: RoutePolicy
metadata:
  name: route-test
  namespace: gwtest
spec:
  ai:
    promptEnrichment:
      prepend:
        - role: SYSTEM
          content: "respond to all questions in French"
        - role: SYSTEM
          content: "Say hello before each response"
    promptGuard:
      request:
        customResponse:
          message: "Rejected due to inappropriate content"
          statusCode: 400
        regex:
          matches:
          - pattern: "credit card"
      response:
        regex:
          builtins:
            - PHONE_NUMBER
            - EMAIL
            - SSN
            - CREDIT_CARD

The APIs follow the required vs. optional API conventions defined in https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#optional-vs-required:

Optional fields:

• Have omitempty set
• Are pointer type in the Go definition (e.g. AwesomeFlag *SomeFlag) or have a built-in nil value (e.g. maps and slices).

Required fields:

• Do not have omitempty set
• They are not a pointer type in the Go definition (e.g. AnotherFlag SomeFlag).

Code changes

Plugin and extauth changes will be implemented in a follow up. This PR only introduces the AI APIs.

CI changes

NONE

Docs changes

NONE

Context

Enhancement Proposal: #10495

Testing steps

Manually validated CR validation by install helm chart and applying example configs.

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

[EItanya]

Overall this make sense to me. I have some nits about structure, but my big takeaways are:

1. What's the decision process between pointer and non-pointer types throughout the API, it's not clear to me.
2. Why are we using iota for enums in some place, and string aliases in others?
3. The AIRouteOptions need more validation akin to the upstream.