Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TT-13391] Move upstream OAuth to EE #6684

Merged
merged 52 commits into from
Nov 5, 2024

Conversation

andrei-tyk
Copy link
Contributor

@andrei-tyk andrei-tyk commented Oct 29, 2024

User description

TT-13359
Summary [Upstream Auth] Move upstream auth features to ee folder
Type Story Story
Status In Code Review
Points N/A
Labels -

Description

TASK: https://tyktech.atlassian.net/browse/TT-13359

Related Issue

Motivation and Context

How This Has Been Tested

Screenshots (if appropriate)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Refactoring or add test (improvements in base code or adds test coverage to functionality)

Checklist

  • I ensured that the documentation is up to date
  • I explained why this PR updates go.mod in detail with reasoning why it's required
  • I would like a code coverage CI quality gate exception and have explained why

PR Type

enhancement, other


Description

  • Refactored context handling across multiple files to use httputil instead of ctx.
  • Implemented upstream OAuth provider and middleware for handling OAuth tokens in the ee package.
  • Moved encryption and decryption logic to the crypto package.
  • Added initialization and caching logic for upstream OAuth in the server.
  • Updated event firing to use model.EventMetaDefault for consistency.
  • Added utilities for context data handling and request encoding to events.

PRDescriptionHeader.CHANGES_WALKTHROUGH

Relevant files
Enhancement
14 files
api.go
Refactor context handling and event firing in API gateway

gateway/api.go

  • Removed in-place modification of http.Request context.
  • Replaced ctx package usage with httputil.
  • Updated event firing to use model.EventMetaDefault.
  • +51/-77 
    mw_oauth2_auth.go
    Replace upstream OAuth with no-op middleware                         

    gateway/mw_oauth2_auth.go

  • Removed upstream OAuth implementation.
  • Added a no-op upstream OAuth middleware.
  • +15/-378
    provider.go
    Implement upstream OAuth provider with token caching         

    ee/middleware/upstreamoauth/provider.go

  • Implemented upstream OAuth provider for client credentials and
    password grants.
  • Added token caching and retrieval logic.
  • +317/-0 
    mw_url_rewrite.go
    Refactor URL rewrite middleware to use httputil                   

    gateway/mw_url_rewrite.go

  • Replaced ctx package usage with httputil for context data handling.
  • +21/-20 
    rpc_backup_handlers.go
    Use crypto package for encryption in RPC backup handlers 

    gateway/rpc_backup_handlers.go

    • Moved encryption and decryption logic to crypto package.
    +9/-75   
    middleware.go
    Refactor middleware context handling and event firing       

    gateway/middleware.go

  • Replaced ctx package usage with httputil for context handling.
  • Updated event firing to use model.EventMetaDefault.
  • +10/-22 
    event_system.go
    Refactor event system to use model package                             

    gateway/event_system.go

  • Moved EventMetaDefault to model package.
  • Updated event firing to use model.EventMetaDefault.
  • +11/-34 
    mw_organisation_activity.go
    Refactor organisation activity middleware for context and events

    gateway/mw_organisation_activity.go

  • Replaced ctx package usage with httputil for context data handling.
  • Updated event firing to use model.EventMetaDefault.
  • +12/-9   
    middleware.go
    Implement upstream OAuth middleware for token handling     

    ee/middleware/upstreamoauth/middleware.go

    • Implemented upstream OAuth middleware for handling OAuth tokens.
    +101/-0 
    server.go
    Initialize upstream OAuth caches in server                             

    gateway/server.go

  • Added initialization for upstream OAuth caches.
  • Defined functions for creating OAuth caches.
  • +29/-2   
    helpers.go
    Add AES encryption and decryption helpers                               

    internal/crypto/helpers.go

    • Added functions for encryption and decryption using AES.
    +69/-0   
    context.go
    Add context data handling utilities                                           

    internal/httputil/context.go

  • Added context data handling functions.
  • Defined context keys for request handling.
  • +56/-0   
    event.go
    Add request encoding to event utility                                       

    internal/event/event.go

    • Added function to encode HTTP requests to events.
    +11/-0   
    events.go
    Add EventMetaDefault struct for event metadata                     

    internal/model/events.go

    • Added EventMetaDefault struct for event metadata.
    +8/-0     
    Other
    1 files
    ctx.go
    Remove context key definitions from ctx package                   

    ctx/ctx.go

    • Removed context key definitions and related functions.
    +7/-45   

    💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

    @buger
    Copy link
    Member

    buger commented Oct 29, 2024

    This PR is too huge for one to review 💔

    Additions 934 🙅‍♀️
    Expected ⬇️ 800

    Consider breaking it down into multiple small PRs.

    Check out this guide to learn more about PR best-practices.

    @buger
    Copy link
    Member

    buger commented Oct 29, 2024

    I'm a bot and I 👍 this PR title. 🤖

    Copy link
    Contributor

    github-actions bot commented Oct 29, 2024

    API Changes

    --- prev.txt	2024-11-05 10:03:35.806275818 +0000
    +++ current.txt	2024-11-05 10:03:29.290300396 +0000
    @@ -8360,6 +8360,147 @@
     func (u Provider) Fill(r *http.Request)
         Fill sets the request's HeaderName with AuthValue
     
    +# Package: ./ee/middleware/upstreamoauth
    +
    +package upstreamoauth // import "github.com/TykTechnologies/tyk/ee/middleware/upstreamoauth"
    +
    +
    +CONSTANTS
    +
    +const (
    +	ErrorEventName = "UpstreamOAuthError"
    +	MiddlewareName = "UpstreamOAuth"
    +
    +	ClientCredentialsAuthorizeType = "clientCredentials"
    +	PasswordAuthorizeType          = "password"
    +)
    +
    +VARIABLES
    +
    +var (
    +	CtxGetData = ctxData.Get
    +	CtxSetData = ctxData.Set
    +)
    +
    +FUNCTIONS
    +
    +func BuildMetadataMap(token *oauth2.Token, extraMetadataKeys []string) map[string]interface{}
    +func CreateTokenDataBytes(encryptedToken string, token *oauth2.Token, extraMetadataKeys []string) ([]byte, error)
    +func SetExtraMetadata(r *http.Request, keyList []string, metadata map[string]interface{})
    +
    +TYPES
    +
    +type BaseMiddleware interface {
    +	model.LoggerProvider
    +	FireEvent(name apidef.TykEvent, meta interface{})
    +}
    +    BaseMiddleware is the subset of BaseMiddleware APIs that the middleware
    +    uses.
    +
    +type Cache interface {
    +	// GetToken returns the token from cache or issues a request to obtain it from the OAuth provider.
    +	GetToken(r *http.Request) (string, error)
    +	// ObtainToken issues a request to obtain the token from the OAuth provider.
    +	ObtainToken(ctx context.Context) (*oauth2.Token, error)
    +}
    +
    +type ClientCredentialsClient struct {
    +	// Has unexported fields.
    +}
    +
    +func (cache *ClientCredentialsClient) GetToken(r *http.Request) (string, error)
    +
    +func (cache *ClientCredentialsClient) ObtainToken(ctx context.Context) (*oauth2.Token, error)
    +
    +type ClientCredentialsOAuthProvider struct{}
    +
    +type EventUpstreamOAuthMeta struct {
    +	model.EventMetaDefault
    +	APIID string
    +}
    +    EventUpstreamOAuthMeta is the metadata structure for an upstream OAuth event
    +
    +type Gateway interface {
    +	model.ConfigProvider
    +}
    +    Gateway is the subset of Gateway APIs that the middleware uses.
    +
    +type Middleware struct {
    +	Spec model.MergedAPI
    +	Gw   Gateway
    +
    +	Base BaseMiddleware
    +
    +	// Has unexported fields.
    +}
    +    Middleware implements upstream OAuth middleware.
    +
    +func NewMiddleware(gw Gateway, mw BaseMiddleware, spec model.MergedAPI, ccStorageHandler Storage, pwStorageHandler Storage) *Middleware
    +    NewMiddleware returns a new instance of Middleware.
    +
    +func (m *Middleware) EnabledForSpec() bool
    +    EnabledForSpec checks if streaming is enabled on the config.
    +
    +func (mw *Middleware) FireEvent(r *http.Request, e event.Event, message string, apiId string)
    +    FireEvent emits an upstream OAuth event with an optional custom message.
    +
    +func (m *Middleware) Init()
    +    Init initializes the middleware.
    +
    +func (m *Middleware) Logger() *logrus.Entry
    +    Logger returns a logger with middleware filled out.
    +
    +func (m *Middleware) Name() string
    +    Name returns the name for the middleware.
    +
    +func (m *Middleware) ProcessRequest(_ http.ResponseWriter, r *http.Request, _ interface{}) (error, int)
    +    ProcessRequest will handle upstream OAuth.
    +
    +type OAuthHeaderProvider interface {
    +	// Has unexported methods.
    +}
    +
    +func NewOAuthHeaderProvider(oauthConfig apidef.UpstreamOAuth) (OAuthHeaderProvider, error)
    +
    +type PasswordClient struct {
    +	// Has unexported fields.
    +}
    +
    +func (cache *PasswordClient) GetToken(r *http.Request) (string, error)
    +
    +func (cache *PasswordClient) ObtainToken(ctx context.Context) (*oauth2.Token, error)
    +
    +type PasswordOAuthProvider struct{}
    +
    +type PerAPIClientCredentialsOAuthProvider struct{}
    +
    +type Provider struct {
    +	// Logger is the logger to be used.
    +	Logger *logrus.Entry
    +	// HeaderName is the header name to be used to fill upstream auth with.
    +	HeaderName string
    +	// AuthValue is the value of auth header.
    +	AuthValue string
    +}
    +    Provider implements upstream auth provider.
    +
    +func (u Provider) Fill(r *http.Request)
    +    Fill sets the request's HeaderName with AuthValue
    +
    +type Storage interface {
    +	GetKey(key string) (string, error)
    +	SetKey(string, string, int64) error
    +	Lock(key string, timeout time.Duration) (bool, error)
    +}
    +    Type Storage is a subset of storage.RedisCluster
    +
    +type TokenData struct {
    +	Token         string                 `json:"token"`
    +	ExtraMetadata map[string]interface{} `json:"extra_metadata"`
    +}
    +
    +func UnmarshalTokenData(tokenData string) (TokenData, error)
    +
     # Package: ./gateway
     
     package gateway // import "github.com/TykTechnologies/tyk/gateway"
    @@ -8497,10 +8638,6 @@
     	ECDSASign = "ecdsa"
     )
     const (
    -	UpstreamOAuthErrorEventName = "UpstreamOAuthError"
    -	UpstreamOAuthMiddlewareName = "UpstreamOAuth"
    -)
    -const (
     	ErrOAuthAuthorizationFieldMissing   = "oauth.auth_field_missing"
     	ErrOAuthAuthorizationFieldMalformed = "oauth.auth_field_malformed"
     	ErrOAuthKeyNotFound                 = "oauth.key_not_found"
    @@ -8575,6 +8712,9 @@
     	GlobalRate = ratecounter.NewRateCounter(1 * time.Second)
     )
     var (
    +	EncodeRequestToEvent = event.EncodeRequestToEvent
    +)
    +var (
     	ErrTokenValidationFailed = errors.New("error happened during the access token validation")
     	ErrKIDNotAString         = errors.New("kid is not a string")
     	ErrNoMatchingKIDFound    = errors.New("no matching KID could be found")
    @@ -8649,10 +8789,6 @@
     func CreateStandardSession() *user.SessionState
     func DoCoprocessReload()
     func DurationToMillisecond(d time.Duration) float64
    -func EncodeRequestToEvent(r *http.Request) string
    -    EncodeRequestToEvent will write the request out in wire protocol and encode
    -    it to base64 and store it in an Event object
    -
     func EnsureTransport(host, protocol string) string
         EnsureTransport sanitizes host/protocol pairs and returns a valid URL.
     
    @@ -9095,8 +9231,6 @@
     	Skip           bool
     }
     
    -type ClientCredentialsOAuthProvider struct{}
    -
     type CoProcessEventHandler struct {
     	Spec     *APISpec
     	SpecJSON json.RawMessage
    @@ -9336,13 +9470,7 @@
         EventKeyFailureMeta is the metadata structure for any failure related to a
         key, such as quota or auth failures.
     
    -type EventMetaDefault struct {
    -	Message            string
    -	OriginatingRequest string
    -}
    -    EventMetaDefault is a standard embedded struct to be used with custom event
    -    metadata types, gives an interface for easily extending event metadata
    -    objects
    +type EventMetaDefault = model.EventMetaDefault
     
     type EventTokenMeta struct {
     	EventMetaDefault
    @@ -9358,12 +9486,6 @@
     	UsagePercentage int64  `json:"usage_percentage"`
     }
     
    -type EventUpstreamOAuthMeta struct {
    -	EventMetaDefault
    -	APIID string
    -}
    -    EventUpstreamOAuthMeta is the metadata structure for an upstream OAuth event
    -
     type EventVersionFailureMeta struct {
     	EventMetaDefault
     	Path   string
    @@ -9468,8 +9590,6 @@
     	HostCheckTicker      chan struct{}
     	HostCheckerClient    *http.Client
     	TracerProvider       otel.TracerProvider
    -	// UpstreamOAuthCache is used to cache upstream OAuth tokens
    -	UpstreamOAuthCache UpstreamOAuthCache
     
     	SessionLimiter SessionLimiter
     	SessionMonitor Monitor
    @@ -10385,10 +10505,6 @@
         in compliance with https://tools.ietf.org/html/rfc7009#section-2.1 ToDo:
         set an authentication mechanism
     
    -type OAuthHeaderProvider interface {
    -	// Has unexported methods.
    -}
    -
     type OAuthManager struct {
     	API        *APISpec
     	OsinServer *TykOsinServer
    @@ -10470,10 +10586,6 @@
     
     func (k *OrganizationMonitor) SetOrgSentinel(orgChan chan bool, orgId string)
     
    -type PasswordOAuthProvider struct{}
    -
    -type PerAPIClientCredentialsOAuthProvider struct{}
    -
     type PersistGraphQLOperationMiddleware struct {
     	*BaseMiddleware
     }
    @@ -11374,11 +11486,6 @@
     	Form    map[string]string
     }
     
    -type TokenData struct {
    -	Token         string                 `json:"token"`
    -	ExtraMetadata map[string]interface{} `json:"extra_metadata"`
    -}
    -
     type TraceMiddleware struct {
     	TykMiddleware
     }
    @@ -11622,38 +11729,6 @@
         Enums representing the various statuses for a VersionInfo Path match during
         a proxy request
     
    -type UpstreamOAuth struct {
    -	*BaseMiddleware
    -}
    -    UpstreamOAuth is a middleware that will do basic authentication for upstream
    -    connections. UpstreamOAuth middleware is only supported in Tyk OAS API
    -    definitions.
    -
    -func (OAuthSpec *UpstreamOAuth) EnabledForSpec() bool
    -    EnabledForSpec returns true if the middleware is enabled based on API Spec.
    -
    -func (OAuthSpec *UpstreamOAuth) Name() string
    -    Name returns the name of middleware.
    -
    -func (OAuthSpec *UpstreamOAuth) ProcessRequest(_ http.ResponseWriter, r *http.Request, _ interface{}) (error, int)
    -    ProcessRequest will inject basic auth info into request context so that it
    -    can be used during reverse proxy.
    -
    -type UpstreamOAuthCache interface {
    -	// Has unexported methods.
    -}
    -
    -type UpstreamOAuthProvider struct {
    -	// HeaderName is the header name to be used to fill upstream auth with.
    -	HeaderName string
    -	// AuthValue is the value of auth header.
    -	AuthValue string
    -}
    -    UpstreamOAuthProvider implements upstream auth provider.
    -
    -func (u UpstreamOAuthProvider) Fill(r *http.Request)
    -    Fill sets the request's HeaderName with AuthValue
    -
     type UptimeReportData struct {
     	URL          string
     	RequestTime  int64

    Copy link
    Contributor

    PR Reviewer Guide 🔍

    Here are some key observations to aid the review process:

    ⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
    🧪 No relevant tests
    🔒 No security concerns identified
    ⚡ Recommended focus areas for review

    Code Refactor
    The context handling functions have been refactored to use a centralized utility, which improves maintainability and consistency. However, it's crucial to ensure that the new context utility functions are thoroughly tested, especially in concurrent environments, to avoid potential issues with context data being overwritten or lost during request processing.

    Feature Reduction
    The OAuth middleware has been significantly simplified, removing support for various OAuth flows directly within the gateway. This change could impact users relying on the previous functionality. It's important to document these changes clearly in the release notes and provide guidelines or external tool recommendations for handling OAuth flows outside of the gateway.

    Context Handling
    The refactoring includes changes to how request contexts are manipulated during URL rewriting. Given the complexity of URL rewriting logic, particularly with chained rewrites and variable extractions, it's crucial to ensure that context changes are propagated correctly without unintended side effects.

    Copy link
    Contributor

    PR Code Suggestions ✨

    Explore these optional code suggestions:

    CategorySuggestion                                                                                                                                    Score
    Possible bug
    Add validation for empty OAuth token to prevent invalid authorization

    Ensure that the ProcessRequest method handles the scenario where the
    provider.getOAuthToken might return an empty token, which could lead to setting an
    invalid authorization header.

    ee/middleware/upstreamoauth/middleware.go [81-86]

     payload, err := provider.getOAuthToken(r, m)
     if err != nil {
         return fmt.Errorf("failed to get OAuth token: %w", err), http.StatusInternalServerError
     }
    +if payload == "" {
    +    return fmt.Errorf("empty OAuth token received"), http.StatusInternalServerError
    +}
     upstreamOAuthProvider.AuthValue = payload
    Suggestion importance[1-10]: 9

    Why: This suggestion addresses a potential bug by ensuring that an empty OAuth token does not lead to setting an invalid authorization header, thus enhancing the reliability of the authentication process.

    9
    Add error handling for cache retrieval operations

    Implement error handling for the retryGetKeyAndLock function to handle potential nil
    errors from the cache operations, ensuring stability and robustness.

    ee/middleware/upstreamoauth/provider.go [208-210]

     token, err = cache.GetKey(cacheKey)
    -if err == nil {
    -    return token, nil
    +if err != nil {
    +    return "", fmt.Errorf("error retrieving token: %w", err)
     }
    +return token, nil
    Suggestion importance[1-10]: 8

    Why: This suggestion enhances the robustness of the code by adding error handling for cache retrieval operations, which is crucial for preventing unexpected failures.

    8
    Best practice
    Implement nil checks on context retrievals to prevent panics

    Check for nil values when retrieving context values to avoid runtime panics.

    gateway/api.go [3074]

    -key, _ := r.Context().Value(httputil.CacheOptions).(*cacheOptions)
    +key, ok := r.Context().Value(httputil.CacheOptions).(*cacheOptions)
    +if !ok {
    +  // handle error
    +}
    Suggestion importance[1-10]: 8

    Why: The suggestion correctly identifies a potential issue with unchecked type assertions when retrieving context values. Adding a nil check can prevent runtime panics, enhancing the robustness of the code.

    8
    Enhancement
    Add error logging for missing context values to aid in debugging

    Use consistent error handling for missing or invalid context values to ensure
    robustness.

    gateway/api.go [3074]

    -key, _ := r.Context().Value(httputil.CacheOptions).(*cacheOptions)
    +key, ok := r.Context().Value(httputil.CacheOptions).(*cacheOptions)
    +if !ok {
    +  log.Errorf("CacheOptions not found in context")
    +  return
    +}
    Suggestion importance[1-10]: 7

    Why: This suggestion improves error handling by adding logging for missing context values, which can be helpful for debugging. While not critical, it enhances code maintainability and observability.

    7
    Remove redundant code that sets the header name twice

    Avoid setting the header name twice in the ProcessRequest method. This redundancy
    can be removed to enhance code clarity and prevent potential bugs.

    ee/middleware/upstreamoauth/middleware.go [92-97]

    -if provider.headerEnabled(m) {
    -    headerName := provider.getHeaderName(m)
    -    if headerName != "" {
    -        upstreamOAuthProvider.HeaderName = headerName
    -    }
    -}
    +// Removed redundant header setting
    Suggestion importance[1-10]: 6

    Why: The suggestion correctly identifies and removes redundant code, which improves clarity and reduces potential bugs related to setting the header name multiple times.

    6
    Maintainability
    Consolidate duplicate functions into a single function for generating OAuth cache keys

    Refactor the generateClientCredentialsCacheKey and generatePasswordOAuthCacheKey
    functions to a single function to avoid code duplication and improve
    maintainability.

    ee/middleware/upstreamoauth/provider.go [187-317]

    -func generateClientCredentialsCacheKey(config apidef.UpstreamOAuth, apiId string) string {
    -    ...
    -}
    -func generatePasswordOAuthCacheKey(config apidef.UpstreamOAuth, apiId string) string {
    +func generateOAuthCacheKey(config apidef.UpstreamOAuth, apiId string) string {
         ...
     }
    Suggestion importance[1-10]: 7

    Why: The suggestion improves maintainability by reducing code duplication, making it easier to manage and update the cache key generation logic.

    7

    internal/crypto/helpers.go Dismissed Show dismissed Hide dismissed
    ee/middleware/upstreamoauth/cache.go Outdated Show resolved Hide resolved
    ee/middleware/upstreamoauth/cache.go Outdated Show resolved Hide resolved
    ee/middleware/upstreamoauth/cache.go Outdated Show resolved Hide resolved
    ee/middleware/upstreamoauth/cache.go Outdated Show resolved Hide resolved
    - moved providers in separate fies
    - adjusted signature of functions to be less verbose and more receiver-focused
    - renamed a few variables/parameters with better choices
    Copy link

    sonarcloud bot commented Nov 5, 2024

    Quality Gate Failed Quality Gate failed

    Failed conditions
    0.0% Coverage on New Code (required ≥ 80%)
    C Reliability Rating on New Code (required ≥ A)

    See analysis details on SonarCloud

    Catch issues before they fail your Quality Gate with our IDE extension SonarLint

    @andrei-tyk andrei-tyk merged commit fa63dbe into master Nov 5, 2024
    34 of 40 checks passed
    @andrei-tyk andrei-tyk deleted the TT-13391-move-upstream-o-auth-to-ee branch November 5, 2024 10:38
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Projects
    None yet
    Development

    Successfully merging this pull request may close these issues.

    4 participants