From 99aceb9c20b40e906f47edfc238a577ef422fe1d Mon Sep 17 00:00:00 2001 From: Albert Snow Date: Mon, 22 Jul 2024 12:29:59 -0500 Subject: [PATCH 1/9] adding additional line to clarify --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index ac413b9ac..0be54577d 100644 --- a/README.md +++ b/README.md @@ -254,6 +254,9 @@ python cfg_help.py get The file should have detailed information about each of the configuration variables. **Remember to fill out the new configuration file!** +Once you have done so, you can run `alembic upgrade head` to generate the tables needed +to run fence. + #### Other Configuration Notes * Fence will look for configuration files from a list of search directories ( From ffcf79f7c9e85349cb8ed66c8e7af7e0f1da49db Mon Sep 17 00:00:00 2001 From: Albert Snow Date: Mon, 22 Jul 2024 12:45:04 -0500 Subject: [PATCH 2/9] moving files around --- README.md | 549 +----------------- docs/{ => azure}/azure_architecture.md | 22 +- docs/{ => azure}/azure_devops_pipeline.md | 10 +- .../azure_devops_pipeline_config_1.png | Bin .../azure_devops_pipeline_config_2.png | Bin .../azure_devops_pipeline_config_3.png | Bin docs/{ => misc}/dbgap_info.md | 0 docs/{ => misc}/fence-create-deprecated.md | 0 .../fence_multifactor_authentication_guide.md | 0 docs/{ => misc}/fence_shibboleth.md | 4 +- docs/{ => misc}/ga4gh_passports.md | 16 +- docs/{ => misc}/google_architecture.md | 16 +- docs/{ => misc}/local_multi_fence.md | 0 docs/{ => misc}/quickstart_helm.md | 0 docs/{ => misc}/register.md | 0 docs/{ => misc}/user.yaml_guide.md | 4 +- docs/{ => misc}/usersync.md | 2 +- 17 files changed, 40 insertions(+), 583 deletions(-) rename docs/{ => azure}/azure_architecture.md (93%) rename docs/{ => azure}/azure_devops_pipeline.md (87%) rename docs/{ => azure}/azure_devops_pipeline_config_1.png (100%) rename docs/{ => azure}/azure_devops_pipeline_config_2.png (100%) rename docs/{ => azure}/azure_devops_pipeline_config_3.png (100%) rename docs/{ => misc}/dbgap_info.md (100%) rename docs/{ => misc}/fence-create-deprecated.md (100%) rename docs/{ => misc}/fence_multifactor_authentication_guide.md (100%) rename docs/{ => misc}/fence_shibboleth.md (95%) rename docs/{ => misc}/ga4gh_passports.md (91%) rename docs/{ => misc}/google_architecture.md (98%) rename docs/{ => misc}/local_multi_fence.md (100%) rename docs/{ => misc}/quickstart_helm.md (100%) rename docs/{ => misc}/register.md (100%) rename docs/{ => misc}/user.yaml_guide.md (98%) rename docs/{ => misc}/usersync.md (99%) diff --git a/README.md b/README.md index 0be54577d..9b18c29dc 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ Fence is a core service of the Gen3 stack that has multiple capabilities: ## Contents 1. [API Documentation](#API-documentation) -1. [Terminologies](#Terminologies) +1. [Terminologies](./docs/introduction/terminology.md) 1. [Identity Providers](#identity-providers) 1. [OIDC & OAuth2](#oidc--oauth2) 1. [Accessing Data](#accessing-data) @@ -35,59 +35,6 @@ Fence is a core service of the Gen3 stack that has multiple capabilities: YAML file for the OpenAPI documentation is found in the `openapis` folder (in the root directory); see the README in that folder for more details. -## Terminologies - -### AuthN - -Authentication - establishes "who you are" with the application through communication with an [Identity Provider](#IdP). - -### AuthZ - -Authorization - establishes "what you can do" and "which resources you have access to" within the application. - -### IdP - -Identity Provider - the service that lets a user login and provides the identity of the user to downstream services. Examples: Google login, University login, NIH Login. - -### Auth broker - -An interface which enables a user to authenticate using any of multiple IdPs. - -### OAuth2 - -A widely used AuthZ protocol for delegating access to an application to use resources on behalf of a user. - -https://tools.ietf.org/html/rfc6749 - -https://oauth.net/2/ - -#### Client - -OAuth 2.0 Client - An application which makes requests for protected resources (on a resource server) on behalf of a resource owner (end-user) and with the resource owner's authorization. - -#### Auth Server - -OAuth 2.0 Authorization Server - A server which issues access tokens to the client after successfully authenticating the resource owner and obtaining authorization. - -#### Access Token - -A string, issued by the auth server to the client, representing authorization credentials used to access protected resources (on a resource server). - -### OIDC - -OpenID Connect - an extension of OAuth2 which provides an AuthN layer on top of the OAuth 2.0 AuthZ layer. It introduced a new type of token, the id token, that is specifically designed to be consumed by clients to get the identity information of the user. - -http://openid.net/specs/openid-connect-core-1_0.html - -#### OP - -OpenID Provider - an OAuth 2.0 Authentication Server which also implements OpenID Connect. - -#### RP - -Relying Party - an OAuth 2.0 Client which uses (requests) OpenID Connect. - - ## Identity Providers @@ -95,7 +42,7 @@ Fence can be configured to support different Identity Providers (IdPs) for AuthN At the moment, supported IDPs include: - Google -- [Shibboleth](docs/fence_shibboleth.md) +- [Shibboleth](docs/misc/fence_shibboleth.md) - NIH iTrust - InCommon - eduGAIN @@ -106,81 +53,6 @@ At the moment, supported IDPs include: - ORCID - RAS -## OIDC & OAuth2 - -Fence acts as a central broker that supports multiple IdPs. -At the same time, it acts as an IdP itself. -In that sense, `fence` is both an `RP` and an `OP`. - -### Fence as RP - -Example: - -- Google IAM is the OpenID Provider (OP) -- Fence is the Relying Party (RP) -- Google Calendar API is the resource provider - -### Fence as OP - -- Fence is the OpenID Provider (OP) -- A third-party application is the Relying Party (RP) -- Gen3 microservices (e.g. [`sheepdog`](https://github.com/uc-cdis/sheepdog)) are resource providers - -### Example Flows - -Note that the `3rd Party App` acts as the `RP` in these examples. - -[//]: # (See /docs folder for README on how to regenerate these sequence diagrams) - -#### Flow: Client Registration - -![Client Registration](docs/images/seq_diagrams/client_registration.png) - -#### Flow: OpenID Connect - -In the following flow, Fence and the IdP together constitute an `OP`. -Fence, by itself, acts as an OAuth 2.0 Auth Server; the IdP enables the additional implementation of OIDC (by providing AuthN). From an OIDC viewpoint, therefore, Fence and the IdP can be abstracted into one `OP`. - -![OIDC Flow](docs/images/seq_diagrams/openid_connect_flow.png) - -If the third-party application doesn't need to use any Gen3 resources (and just -wants to authenticate the user), they can just get -needed information in the `ID token` after the handshake is finished . - -#### Flow: Using Tokens for Access - -If a third-party application wants to use Gen3 resources like -`fence`/`sheepdog`/`peregrine`, they call those services with an `Access Token` -passed in an `Authorization` header. - -In the following flow, `3rd Party App` is the `RP`; `Protected Endpoint` is an endpoint of a Gen3 Resource (the `microservice`), and both of these are part of a `resource server`; and `Fence` is the `OP`. Here, importantly, `Fence` may be interfacing with another IdP _or_ with another `Fence` instance in order to implement the OIDC layer. Either way, note that the `Fence` blob in this diagram actually abstracts Fence in concert with some IdP, which may or may not also be (a different instance of) Fence. - -![Using Access Token](docs/images/seq_diagrams/token_use_for_access.png) - -#### Flow: Refresh Token Use - -![Using Refresh Token](docs/images/seq_diagrams/refresh_token_use.png) - -#### Flow: Refresh Token Use (Token is Expired) - -![Using Expired Refresh Token](docs/images/seq_diagrams/refresh_token_use_expired.png) - -#### Flow: Multi-Tenant Fence - -The following diagram illustrates the case in which one fence instance -uses another fence instance as its identity provider. - -A use case for this is when we setup a fence instance that uses NIH login as the IdP. Here, we go through a detailed approval process in NIH. Therefore we would like to do it only once for a single lead Fence instance, and then allow other fence instances to simply redirect to use the lead Fence as an IdP for logging in via NIH. - -In the following flow, `Fence (Client Instance)` is an OP relative to `OAuth Client`, but an RP relative to `Fence (IDP)`. - -![Multi-Tenant Flow](docs/images/seq_diagrams/multi-tenant_flow.png) - -#### Notes - -See the [OIDC specification](http://openid.net/specs/openid-connect-core-1_0.html) for more details. -Additionally, see the [OAuth2 specification](https://tools.ietf.org/html/rfc6749). - ## Access Control / Authz Currently fence works with another Gen3 service named @@ -216,421 +88,6 @@ Fence has the ability to request a specific file by its GUID (globally unique id Whereas pre-signed URL is a cloud agnostic solution, services and tools on Google Cloud Platform prefer to use Google's concept of a "Service Account". Because of that, Fence provides a few more methods to access data in Google. -See [Fence and Google](docs/google_architecture.md) for more details on data access methods specific to Google. - -## Setup - -### Install Requirements and Fence - -Install [Poetry](https://python-poetry.org/docs/#installation). - -```bash -# Install Fence and dependencies -poetry install -``` - -### Create Configuration File - -Fence requires a configuration file to run. We have a command line -utility to help you create one based on a default configuration. - -The configuration file itself will live outside of this repo (to -prevent accidentally checking in sensitive information like database passwords). - -To create a new configuration file from the default configuration: - -```bash -python cfg_help.py create -``` - -This file will be placed in one of the default search directories for Fence. - -To get the exact path where the new configuration file was created, use: - -```bash -python cfg_help.py get -``` - -The file should have detailed information about each of the configuration -variables. **Remember to fill out the new configuration file!** - -Once you have done so, you can run `alembic upgrade head` to generate the tables needed -to run fence. - -#### Other Configuration Notes - -* Fence will look for configuration files from a list of search directories ( -which are currently defined in `fence/settings.py`.) -* For more configuration options (such as having multiple different config -files for development), see the `cfg_help.py` file. - -### Set Up Databases - -The tests clear out the database every time they are run. If you want -to keep a persistent database for manual testing and general local usage, -create a second test database with a different name: - -> NOTE: Requires a minimum of Postgres v9.4 (because of `JSONB` types used) - -```bash -# Create test database(s). -# This one is for automated tests, which clear the database after running; -# `tests/test_settings.py` should have `fence_test_tmp` in the `DB` variable. -psql -U test postgres -c 'create database fence_test_tmp' -userdatamodel-init --db fence_test_tmp -# This one is for manual testing/general local usage; Your config -# should have `fence_test` in the `DB` variable. -psql -U test postgres -c 'create database fence_test' -userdatamodel-init --db fence_test --username test --password test -``` - -### Keypair Configuration - -Fence uses RSA keypairs to sign and allow verification of JWTs that it issues. -When the application is initialized, Fence loads in keypair files from the -`keys` directory. To store keypair files, use the following procedure: - - Create a subdirectory in the `fence/keys` directory, named with a - unique identifier, preferably a timestamp in ISO 8601 format of when - the keys are created. The name of the directory is used for the `kid` - (key ID) for those keys; the default (assuming the directory is named - with an ISO timestamp) looks like this: - - fence_key_2018-05-01T14:00:00Z - - - Generate a private and public keypair following the RSA 256 algorithm - and store those in that directory. The key files must be named - `jwt_public_key.pem` and `jwt_private_key.pem`. - -To generate a keypair using `openssl`: -```bash -# Generate the private key. -openssl genpkey -algorithm RSA -out jwt_private_key.pem -pkeyopt rsa_keygen_bits:2048 - -# Generate the public key. -openssl rsa -pubout -in jwt_private_key.pem -out jwt_public_key.pem - -# Depending on the `openssl` distribution, you may find these work instead: -# -# openssl rsa -out private_key.pem 2048 -# openssl rsa -in private_key.pem -pubout -out public_key.pem -``` -It's not a bad idea to confirm that the files actually say `RSA PRIVATE KEY` -and `PUBLIC KEY` (and in fact Fence will require that the private key files it -uses actually say "PRIVATE KEY" and that the public keys do not). - -Files containing public/private keys should have this format (the format used -by `openssl` for generating RSA keys): -``` ------BEGIN PUBLIC KEY----- -... [key is here] ... ------END PUBLIC KEY----- -``` -If a key is not in this format, then `PyJWT` will raise errors about not being -able to read the key. - -Fence will use the first keypair in the list to sign the tokens it issues -through OAuth. - - -### Create User Access File - -You can setup user access via admin fence script providing a user yaml file -Example user yaml: -``` -cloud_providers: {} -groups: {} -users: - userA@gmail.com: - projects: - - auth_id: project_a - privilege: [read, update, create, delete] - - auth_id: project_b - privilege: [read] - userB@gmail.com: - projects: - - auth_id: project_b - privilege: [read] -``` -Example sync command: - -```bash -fence-create sync --yaml user.yaml -``` - -### Register OAuth Client - -When you want to build an application that uses Gen3 resources on behalf of a user, you should register an OAuth client for this app. -Fence right now exposes client registration via admin CLI, because the Oauth2 client for a Gen3 commons needs approval from the sponsor of the commons. If you are an external developer, you should submit a support ticket. - -As a Gen3 commons administrator, you can run following command for an approved client: -```bash -fence-create client-create --client CLIENT_NAME --urls OAUTH_REDIRECT_URL --username USERNAME -``` -This command should output a tuple of `(client_id, client_secret)` which must be -saved by the OAuth client to use with -`fence`. - -## Quickstart with Helm - -You can now deploy individual services via Helm! -Please refer to the Helm quickstart guide HERE (https://github.com/uc-cdis/fence/blob/master/docs/quickstart_helm.md) - -## Token management - -Fence utilizes [OpenID Connect](#OIDC) to generate tokens -for clients. It can also provide tokens directly to a user. - -Clients and users may then use those tokens with other -Gen3 Data Commons services to access protected endpoints that require specific permissions. - -We use JSON Web Tokens (JWTs) as the format for all tokens of the following types: - -- OIDC ID token: this token is used by the OIDC client to get a user's identity from the token content -- OIDC access token: this token can be sent to Gen3 services via bearer header and get protected resources. -- OIDC refresh token: this token can be sent to fence to request a new access / id token. - - - -### JWT Information - -#### Example ID Token - -``` -{ - "sub": "7", - "azp": "test-client", - "pur": "id", - "aud": [ - "openid", - "user", - "test-client" - ], - "context": { - "user": { - "is_admin": false, - "name": "test", - "projects": { - "phs000178": [ - "read", - "update", - "create", - "delete", - "read-storage" - ] - }, - "google": { - "linked_google_account": "somebody@example.com" - } - } - }, - "iss": "https://commons.org", - "jti": "3ae2910b-0294-43dc-af2a-03fd60082aef", - "exp": 1516983302, - "iat": 1516982102, - "auth_time": 1516982102 -} -``` - -#### Example Access Token - -``` -{ - "sub": "7", - "azp": "test-client", - "pur": "access", - "aud": [ - "openid", - "user", - "test-client" - ], - "context": { - "user": { - "is_admin": false, - "name": "test", - "projects": { - "phs000178": [ - "read", - "update", - "create", - "delete", - "read-storage" - ] - }, - "google": { - "proxy_group": "abcdefgh123456", - "linked_google_account": "somebody@example.com" - } - } - }, - "iss": "https://commons.org", - "jti": "2e6ade06-5afb-4ce7-9ab5-e206225ce291", - "exp": 1516983302, - "iat": 1516982102 -} -``` - -#### Example Refresh Token - -``` -{ - "sub": "7", - "azp": "test-client", - "pur": "refresh", - "aud": [ - "openid", - "user", - "test-client" - ], - "iss": "https://commons.org", - "jti": "c72e5573-39fa-4391-a445-191e370b7cc5", - "exp": 1517010902, - "iat": 1516982102 -} -``` - -## fence-create: Automating common tasks with a command line interface - -fence-create is a command line utility that is bundled with fence and allows you to automate some commons tasks within fence. For the latest and greatest run the command `fence-create --help`. - -WARNING: fence-create directly modifies the database in some cases and may circumvent security checks (most of these utilities are used for testing). BE CAREFUL when you're running these commands and make sure you know what they're doing. - - -### Register Internal Oauth Client - -As a Gen3 commons administrator, if you want to create an oauth client that skips user consent step, use the following command: - -```bash -fence-create client-create --client CLIENT_NAME --urls OAUTH_REDIRECT_URL --username USERNAME --auto-approve (--expires-in 30) -``` - -The optional `--expires-in` parameter allows specifying the number of days until this client expires. - -### Register an Implicit Oauth Client - -As a Gen3 commons administrator, if you want to create an implicit oauth client for a webapp: - -```bash -fence-create client-create --client fancywebappname --urls 'https://betawebapp.example/fence -https://webapp.example/fence' --public --username fancyapp --grant-types authorization_code refresh_token implicit -``` - -If there are more than one URL to add, use space to delimit them like this: - -```bash -fence-create client-create --urls 'https://url1/' 'https://url2/' --client ... -``` - -To specify allowed scopes, use the `allowed-scopes` argument: -```bash -fence-create client-create ... --allowed-scopes openid user data -``` - -### Register an Oauth Client for a Client Credentials flow - -The OAuth2 Client Credentials flow is used for machine-to-machine communication and scenarios in which typical authentication schemes like username + password do not make sense. The system authenticates and authorizes the app rather than a user. See the [OAuth2 specification](https://www.rfc-editor.org/rfc/rfc6749#section-4.4) for more details. - -As a Gen3 commons administrator, if you want to create an OAuth client for a client credentials flow: - -```bash -fence-create client-create --client CLIENT_NAME --grant-types client_credentials (--expires-in 30) -``` - -This command will return a client ID and client secret, which you can then use to obtain an access token: - -```bash -curl --request POST https://FENCE_URL/oauth2/token?grant_type=client_credentials -d scope="openid user" --user CLIENT_ID:CLIENT_SECRET -``` - -The optional `--expires-in` parameter allows specifying the number of *days* until this client expires. The recommendation is to rotate credentials with the `client_credentials` grant at least once a year (see [Rotate client credentials](#rotate-client-credentials) section). - -NOTE: In Gen3, you can grant specific access to a client the same way you would to a user. See the [user.yaml guide](https://github.com/uc-cdis/fence/blob/master/docs/user.yaml_guide.md) for more details. - -NOTE: Client credentials tokens are not linked to a user (the claims contain no `sub` or `context.user.name` like other tokens). Some Gen3 endpoints that assume the token is linked to a user, or whose logic require there being a user, do not support them. For an example of how to adapt an endpoint to support client credentials tokens, see [here](https://github.com/uc-cdis/requestor/commit/a5078fae27fa258ac78045cf2bb89cb2104f53cf). For an example of how to explicitly reject client credentials tokens, see [here](https://github.com/uc-cdis/requestor/commit/0f4974c25343d2185c7cdb48dcdeb58f97800672). - -### Modify OAuth Client - -```bash -fence-create client-modify --client CLIENT_NAME --urls http://localhost/api/v0/oauth2/authorize -``` - -That command should output any modifications to the client. Similarly, multiple URLs are -allowed here too. - -Add `--append` argument to add new callback urls or allowed scopes to existing client (instead of replacing them) using `--append --urls` or `--append --allowed-scopes` -```bash -fence-create client-modify --client CLIENT_NAME --urls http://localhost/api/v0/new/oauth2/authorize --append (--expires-in 30) -``` - -### Rotate client credentials - -Use the `client-rotate` command to receive a new set of credentials (client ID and secret) for a client. The old credentials are NOT deactivated and must be deleted or expired separately (see [Delete Expired OAuth Clients](#delete-expired-oauth-clients) section). This allows for a rotation without downtime. - -```bash -fence-create client-rotate --client CLIENT_NAME (--expires-in 30) -``` - -Note that the `usersync` job must be run after rotating the credentials so that the new client ID is granted the same access as the old one. - -### Delete OAuth Client - -```bash -fence-create client-delete --client CLIENT_NAME -``` -That command should output the result of the deletion attempt. - -### Delete Expired OAuth Clients - -```bash -fence-create client-delete-expired -``` - -To post a warning in Slack about any clients that expired or are about to expire: - -```bash -fence-create client-delete-expired --slack-webhook --warning-days -``` - - -### List OAuth Clients - -```bash -fence-create client-list -``` -That command should output the full records for any registered OAuth clients. - -### Set up for External Buckets on Google - -```bash -fence-create link-external-bucket --bucket-name demo-bucket -fence-create link-bucket-to-project --bucket_id demo-bucket --bucket_provider google --project_auth_id test-project -``` - -The link-external-bucket returns an email for a Google group which needs to be added to access to the bucket `demo-bucket`. - -### Notify users who are blocking service account registration - -```bash -fence-create notify-problem-users --emails ex1@gmail.com ex2@gmail.com --auth_ids test --google_project_id test-google -``` - -`notify-problem-users` emails users in the provided list (can be fence user email or linked google email) who do not have access to any of the auth_ids provided. Also accepts a `check_linking` flag to check that each user has linked their google account. - -## Default Expiration Times in Fence - -Table contains various artifacts in fence that have temporary lifetimes and their default values. +See [Fence and Google](docs/misc/google_architecture.md) for more details on data access methods specific to Google. -> NOTE: "SA" in the below table stands for Service Account -| Name | Lifetime | Extendable? | Maximum Lifetime | Details -|--------------------------------------|--------------|-------------|-----------------------|------------------------------------------------------------------------------------------| -| Access Token | 20 minutes | TRUE | Life of Refresh Token | | -| Refresh Token | 30 days | FALSE | N/A | | -| User's SA Account Access | 7 days | TRUE | N/A | Access to data (e.g. length it stays in the proxy group). Can optionally provide an expiration less than 7 days | -| User's Google Account Access | 1 day | TRUE | N/A | After AuthN, how long we associate a Google email with the given user. Can optionally provide an expiration less than 1 day | -| User's Google Account Linkage | Indefinite | N/A | N/A | Can optionally provide an expiration less than 1 hour | -| Google Signed URL | Up to 1 hour | FALSE | N/A | Can optionally provide an expiration less than 1 hour | -| AWS Signed URL | Up to 1 hour | FALSE | N/A | Obtained by an oauth client through /credentials/google | -| Client SA (for User) Key | 10 days | FALSE | N/A | Obtained by the user themselves for temp access. Can optionally provide an expiration less than 10 days | -| User Primary SA Key | 10 days | FALSE | N/A | Used for Google URL signing | -| User Primary SA Key for URL Signing | 30 days | FALSE | N/A | | -| Sliding Session Window | 15 minutes | TRUE | 8 hours | access_token cookies get generated automatically when expired if session is still active | diff --git a/docs/azure_architecture.md b/docs/azure/azure_architecture.md similarity index 93% rename from docs/azure_architecture.md rename to docs/azure/azure_architecture.md index b47c19a41..955bde7ac 100755 --- a/docs/azure_architecture.md +++ b/docs/azure/azure_architecture.md @@ -11,7 +11,7 @@ You can review how `fence` works with [Azure Blob Storage](#Azure-Blob-Storage) ### Azure Blob Storage -![Azure Blob Storage with Fence](./images/m_fence_azure_blob_storage.png) +![Azure Blob Storage with Fence](../images/m_fence_azure_blob_storage.png) The diagram shows 2 separate workflows in order for `fence` to interact with [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction): @@ -44,7 +44,7 @@ You can use the Azure Blob Storage client to connect to Azure Blob Storage, and #### Configuration Details -You can update the [Fence config.yaml](../fence/config-default.yaml) to include the following values: +You can update the [Fence config.yaml](../../fence/config-default.yaml) to include the following values: Name | Value | Description ------ | ------|---------- @@ -64,7 +64,7 @@ For example, when you index the file (e.g. using the [gen3sdk](https://github.co So if you navigate to `https://mydatacommons/files/guid` (assuming that the metadata is already setup), you can click on the file to download which will make the call to get the appropriate signed URL. -![Presigned URL](./images/m_fence_presigned_url.png) +![Presigned URL](../images/m_fence_presigned_url.png) * If you index the file with a URL for a blob in a **public** Azure Blob Storage Container and the `AZ_BLOB_CREDENTIALS` are set to `'*'`, then the **non-signed** converted indexed URL will be used (e.g. `https://.blob.core.windows.net//some/path/to/file.txt`) > You need to replace the URL such as `https://.blob.core.windows.net//some/path/to/file.txt` with `az://.blob.core.windows.net//some/path/to/file.txt` upon submitting the record to `indexd`. @@ -83,7 +83,7 @@ You can use [user delegation SAS tokens](https://docs.microsoft.com/en-us/rest/a ### Azure Active Directory -![Azure AD with Fence](./images/m_fence_azure_AD.png) +![Azure AD with Fence](../images/m_fence_azure_AD.png) The diagram shows 3 separate workflows in order for `fence` to interact with Azure AD: @@ -104,16 +104,16 @@ Also note that there's alternatives that could be considered for [future develop 1. [Create](https://docs.microsoft.com/en-us/azure/data-explorer/provision-azure-ad-app) AAD Application 2. Add a redirect URL * The application needs to have redirect URL that is the FDQN of commons appended with `(commons fdqn)/user/login/microsoft/login`. -![Add Redirect URI](./images/m_fence_azure_AD_app_registration_1.png) +![Add Redirect URI](../images/m_fence_azure_AD_app_registration_1.png) 3. Set a secret for the AAD application -![Set the Client Secret](./images/m_fence_azure_AD_app_registration_2.png) +![Set the Client Secret](../images/m_fence_azure_AD_app_registration_2.png) 4. Retrieve the `client id` of the AAD application -![Retrieve client ID](./images/m_fence_azure_AD_app_registration_3.png) -5. Update [fence-config.yaml](../fence/config-default.yaml) +![Retrieve client ID](../images/m_fence_azure_AD_app_registration_3.png) +5. Update [fence-config.yaml](../../fence/config-default.yaml) * Set the `microsoft_client_id` to be the `client_id` in step 4. * Set the `microsoft_client_secret` to be the secret value in step 3. - * Make sure the `BASE_URL` in [fence-config.yaml](../fence/config-default.yaml) is correct. - * Make sure the `redirect_url` in [fence-config.yaml](../fence/config-default.yaml) is `{{BASE_URL}}/login/microsoft/login/` is matches the redirect URL (`(commons fdqn)/user/login/microsoft/login`) in step 2 + * Make sure the `BASE_URL` in [fence-config.yaml](../../fence/config-default.yaml) is correct. + * Make sure the `redirect_url` in [fence-config.yaml](../../fence/config-default.yaml) is `{{BASE_URL}}/login/microsoft/login/` is matches the redirect URL (`(commons fdqn)/user/login/microsoft/login`) in step 2 6. Restart `fence` service with the updated secrets #### User Yaml Setup @@ -122,7 +122,7 @@ Also note that there's alternatives that could be considered for [future develop It's helpful to understand some of the [Arborist terms and definitions](https://github.com/uc-cdis/arborist#terminology-and-definitions), which covers **action**, **permission**, **role**, **resource**, **policy**, and **group**. -Further, it's helpful to understand the Arborist options for [configuring access](https://github.com/uc-cdis/arborist#configuring-access). You can see an example of granting **users** and **groups** access and more details in the [user.yaml guide](./user.yaml_guide.md). +Further, it's helpful to understand the Arborist options for [configuring access](https://github.com/uc-cdis/arborist#configuring-access). You can see an example of granting **users** and **groups** access and more details in the [user.yaml guide](../misc/user.yaml_guide.md). At a high level, this setup involves a couple steps: diff --git a/docs/azure_devops_pipeline.md b/docs/azure/azure_devops_pipeline.md similarity index 87% rename from docs/azure_devops_pipeline.md rename to docs/azure/azure_devops_pipeline.md index b0ff95d82..7e7be9fd8 100755 --- a/docs/azure_devops_pipeline.md +++ b/docs/azure/azure_devops_pipeline.md @@ -1,12 +1,12 @@ # Azure DevOps Build Pipeline -The purpose of this [Azure DevOps Pipeline](../azure-devops-pipeline.yaml) is to build `fence`, run a test suite, and then push the `fence` container into an [Azure Container Registry](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-get-started-portal). +The purpose of this [Azure DevOps Pipeline](../../azure-devops-pipeline.yaml) is to build `fence`, run a test suite, and then push the `fence` container into an [Azure Container Registry](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-get-started-portal). ## Getting Started If you don't already have access, you can use the free sign up with [Azure Devops](https://docs.microsoft.com/en-us/azure/devops/pipelines/get-started/pipelines-sign-up?view=azure-devops). -You can also import the [pipeline](../azure-devops-pipeline.yaml), see these [doc notes](https://docs.microsoft.com/en-us/azure/devops/pipelines/get-started/clone-import-pipeline?view=azure-devops&tabs=yaml#export-and-import-a-pipeline) as a guide. +You can also import the [pipeline](../../azure-devops-pipeline.yaml), see these [doc notes](https://docs.microsoft.com/en-us/azure/devops/pipelines/get-started/clone-import-pipeline?view=azure-devops&tabs=yaml#export-and-import-a-pipeline) as a guide. ### Setup Azure Container Registry @@ -103,7 +103,7 @@ First, make sure you have already [imported your Azure DevOps Pipeline](https:// Click on the pipeline and then click edit, which will let you update the variables in the Azure DevOps pipeline: -![Click on Variables](./azure_devops_pipeline_config_1.png) +![Click on Variables](azure_devops_pipeline_config_1.png) Variable Name | Description ------ | ------ @@ -117,8 +117,8 @@ GIT_REPO_TAG | This is the tag to use for the `fence` git repository, with a def After updating the variables, be sure to click **save**: -![Save updated variables](./azure_devops_pipeline_config_2.png) +![Save updated variables](azure_devops_pipeline_config_2.png) You can run the pipeline to validate the `fence` build and push to ACR. -![Run the pipeline](./azure_devops_pipeline_config_3.png) \ No newline at end of file +![Run the pipeline](azure_devops_pipeline_config_3.png) \ No newline at end of file diff --git a/docs/azure_devops_pipeline_config_1.png b/docs/azure/azure_devops_pipeline_config_1.png similarity index 100% rename from docs/azure_devops_pipeline_config_1.png rename to docs/azure/azure_devops_pipeline_config_1.png diff --git a/docs/azure_devops_pipeline_config_2.png b/docs/azure/azure_devops_pipeline_config_2.png similarity index 100% rename from docs/azure_devops_pipeline_config_2.png rename to docs/azure/azure_devops_pipeline_config_2.png diff --git a/docs/azure_devops_pipeline_config_3.png b/docs/azure/azure_devops_pipeline_config_3.png similarity index 100% rename from docs/azure_devops_pipeline_config_3.png rename to docs/azure/azure_devops_pipeline_config_3.png diff --git a/docs/dbgap_info.md b/docs/misc/dbgap_info.md similarity index 100% rename from docs/dbgap_info.md rename to docs/misc/dbgap_info.md diff --git a/docs/fence-create-deprecated.md b/docs/misc/fence-create-deprecated.md similarity index 100% rename from docs/fence-create-deprecated.md rename to docs/misc/fence-create-deprecated.md diff --git a/docs/fence_multifactor_authentication_guide.md b/docs/misc/fence_multifactor_authentication_guide.md similarity index 100% rename from docs/fence_multifactor_authentication_guide.md rename to docs/misc/fence_multifactor_authentication_guide.md diff --git a/docs/fence_shibboleth.md b/docs/misc/fence_shibboleth.md similarity index 95% rename from docs/fence_shibboleth.md rename to docs/misc/fence_shibboleth.md index 2cae15ef5..41db778ac 100644 --- a/docs/fence_shibboleth.md +++ b/docs/misc/fence_shibboleth.md @@ -21,7 +21,7 @@ The `/login/shib` endpoint accepts the query parameter `shib_idp`. Fence checks After the user logs in and is redirected to `/login/shib/login`, we get the `eppn` (EduPerson Principal Name) from the request headers to use as username. If the `eppn` is not available, we use the `persistent-id` (or `cn`) instead. -![Shibboleth Login Flow](images/seq_diagrams/shibboleth_flow.png) +![Shibboleth Login Flow](../images/seq_diagrams/shibboleth_flow.png) Notes about the NIH login implementation: - NIH login is used as the default when the `idp` is fence and no `shib_idp` is specified (for backwards compatibility). @@ -32,7 +32,7 @@ Notes about the NIH login implementation: ### In the multi-tenant Fence instance -The [Shibboleth dockerfile](../DockerfileShib) image is at https://quay.io/repository/cdis/fence-shib and is NOT compatible yet with python 3/the latest Fence (for now, use Fence 2.7.x). +The [Shibboleth dockerfile](../../DockerfileShib) image is at https://quay.io/repository/cdis/fence-shib and is NOT compatible yet with python 3/the latest Fence (for now, use Fence 2.7.x). The deployment only includes `revproxy` and `fenceshib`. The Fence configuration enables the `shibboleth` provider: diff --git a/docs/ga4gh_passports.md b/docs/misc/ga4gh_passports.md similarity index 91% rename from docs/ga4gh_passports.md rename to docs/misc/ga4gh_passports.md index 0e6f0da21..235a248cd 100644 --- a/docs/ga4gh_passports.md +++ b/docs/misc/ga4gh_passports.md @@ -25,7 +25,7 @@ References: This shows external DRS Client(s) communicating with Gen3 Framework Services (as a GA4GH DRS Server) and how G3FS interacts with Passport Brokers to validate and verify JWTs. -![Passport and Visa JWT Handling](images/ga4gh/passport_jwt_handling.png) +![Passport and Visa JWT Handling](../images/ga4gh/passport_jwt_handling.png) ## G3FS: Configurable Roles for Data Access @@ -33,11 +33,11 @@ Gen3 Framework Services are capable of acting in many different roles. As data r In order to describe the role of the passport in these various configurations, the following diagrams may help. -![Gen3 as DRS Server](images/ga4gh/gen3_as_drs.png) +![Gen3 as DRS Server](../images/ga4gh/gen3_as_drs.png) -![Gen3 as Client](images/ga4gh/gen3_as_client.png) +![Gen3 as Client](../images/ga4gh/gen3_as_client.png) -![Gen3 as Both](images/ga4gh/gen3_as_client_and_drs_server.png) +![Gen3 as Both](../images/ga4gh/gen3_as_client_and_drs_server.png) ## Performance Improvements @@ -52,15 +52,15 @@ We added a number of things to mitigate the performance impact on researchers' w To illustrate the need for such a cache, see the images below for before and after. -![Before Caching](images/ga4gh/caching_before.png) +![Before Caching](../images/ga4gh/caching_before.png) -![After Caching](images/ga4gh/caching_after.png) +![After Caching](../images/ga4gh/caching_after.png) ## User Identities Different GA4GH Visas may refer to the same subject differently. In order to maintain the known mappings between different representations of the same identity, we are creating an Issuer+Subject to User mapping table. The primary key on this table is the combination of the `iss` and `sub` from JWTs. -![User Identities](images/ga4gh/users.png) +![User Identities](../images/ga4gh/users.png) ## Backend Updates and Expiration @@ -68,6 +68,6 @@ In order to ensure the removal of access at the right time, the cronjobs we have There is an argument here for event-based architecture, but Gen3 does not currently support such an architecture. We are instead extending the support of our cronjobs to ensure expirations occur at the right time. -![Cronjobs and Expirations](images/ga4gh/expiration.png) +![Cronjobs and Expirations](../images/ga4gh/expiration.png) > _All diagrams are originally from an **internal** CTDS Document. The link to that document is [here](https://lucid.app/lucidchart/5c52b868-5cd2-4c6e-b53b-de2981f7da98/edit?invitationId=inv_9a757cb1-fc81-4189-934d-98c3db06d2fc) for internal people who need to edit the above diagrams._ diff --git a/docs/google_architecture.md b/docs/misc/google_architecture.md similarity index 98% rename from docs/google_architecture.md rename to docs/misc/google_architecture.md index b00012ff0..65bc8d7d1 100644 --- a/docs/google_architecture.md +++ b/docs/misc/google_architecture.md @@ -30,7 +30,7 @@ To support the 3 methods of access mentioned above, we have a generic architectu That architecture involves Google's concept of **groups** and use of their **IAM Policies** in the Google Cloud Platform. The following diagram shows the layers between the user themselves and the bucket. -![Google Access Architecture](images/g_architecture.png) +![Google Access Architecture](../images/g_architecture.png) Working backwards from the Google Bucket itself, we have a **Google Bucket Access Group**, which, as you probably guessed, is a Google Group that provides access to the bucket. That group is assigned a **role** on the Google **resource** (the Google Bucket). **Roles** provide a set of permissions (like read privileges). The combinations of those roles on the bucket become the bucket's **Policy**. You can read more about Google's IAM terms and concepts in [their docs](https://cloud.google.com/iam/docs). @@ -46,7 +46,7 @@ Google groups contain **members** (another Google term) and a Google group can b A more representative diagram of the structures that allow users to get access to the buckets may look something like this: -![Representative Google Access Architecture](images/rep_g_architecture.png) +![Representative Google Access Architecture](../images/rep_g_architecture.png) #### User's Proxy Group @@ -169,7 +169,7 @@ In the above script, `google-project-to-bill` is either the `userProject` provid Fence facilitates the creation of Signed URLs to access Google Storage objects. These URLs provide temporary, authenticated, access to anyone with the URL but must be generated by someone who has access. -![Signed URLs](images/signed_urls.png) +![Signed URLs](../images/signed_urls.png) Design Requirements: @@ -195,7 +195,7 @@ This allows clients to manage their temporary credentials without the chance of Each Client Service Account is a member in the User's Proxy Group, meaning it has the same access that the user themselves have. -![Temporary Service Account Credentials](images/g_sa_creds.png) +![Temporary Service Account Credentials](../images/g_sa_creds.png) > WARNING: By default, Google Service Account Keys have an expiration of 10 years. To create a more manageable and secure expiration you must manually "expire" the keys by deleting them with a cronjob (once they are alive longer than a configured expiration). Fence's command line tool `fence-create` has a function for expiring keys that you should run on a schedule. Check out `fence-create google-manage-keys --help` @@ -229,7 +229,7 @@ A user logs into fence with their eRA Commons ID. To get access to data through Google Account Linking is achieved by sending the user through the beginning of the OIDC flow with Google. The user is redirected to a Google Login page and whichever account they successfully log in to becomes linked to their fence identity. -![Google Account Linking](images/g_accnt_link.png) +![Google Account Linking](../images/g_accnt_link.png) We require the user to log in so that we can authenticate them and only link an account they actually own. @@ -239,7 +239,7 @@ Once linked, the user's Google Account is then placed *temporarily* inside their At the moment, the *link* between the User and their Google Account does not expire. The access to data *does* expire though. Explicit refreshing of access must be done by an authenticated user or valid client with those permissions through Fence's API. -![Google Account Linking After Expiration](images/g_accnt_link_2.png) +![Google Account Linking After Expiration](../images/g_accnt_link_2.png) #### Service Account Registration @@ -312,7 +312,7 @@ The Service Accounts are validated first in the cronjob so that if multiple SA's This diagram shows a single Google Project with 3 users (`UserA`, `UserB`, and `UserC`). All of them have already gone through the linking process with fence to associate their Google Account with their fence identity. -![Service Account Registration](images/sa_reg.png) +![Service Account Registration](../images/sa_reg.png) The project service account, `Service Account A`, has been registered for access to a fence `Project` which has data in `Bucket Y`. The service account is given access by placing it *directly in the Google Bucket Access Group*. @@ -326,6 +326,6 @@ The user must request fence `Projects` that the service account should have acce If someone attempting to register `Service Account A` with fence `Projects` that have data in *both* `Bucket X` and `Bucket Y`, registration will fail. Why? Because not every user in the Google Project have access to that data. -![Service Account Registration](images/sa_invalid_reg.png) +![Service Account Registration](../images/sa_invalid_reg.png) --- diff --git a/docs/local_multi_fence.md b/docs/misc/local_multi_fence.md similarity index 100% rename from docs/local_multi_fence.md rename to docs/misc/local_multi_fence.md diff --git a/docs/quickstart_helm.md b/docs/misc/quickstart_helm.md similarity index 100% rename from docs/quickstart_helm.md rename to docs/misc/quickstart_helm.md diff --git a/docs/register.md b/docs/misc/register.md similarity index 100% rename from docs/register.md rename to docs/misc/register.md diff --git a/docs/user.yaml_guide.md b/docs/misc/user.yaml_guide.md similarity index 98% rename from docs/user.yaml_guide.md rename to docs/misc/user.yaml_guide.md index ec9e583eb..893d32045 100644 --- a/docs/user.yaml_guide.md +++ b/docs/misc/user.yaml_guide.md @@ -29,13 +29,13 @@ In a fully deployed Gen3 Commons using [Cloud Automation](https://github.com/uc- } ``` -A template, ready-to-use `user.yaml` file can be found [here](base_user.yaml). +A template, ready-to-use `user.yaml` file can be found [here](../base_user.yaml). When updating your `user.yaml` file, you should use the [`gen3users` CLI](https://github.com/uc-cdis/gen3users#gen3users) to validate it before use. ## Format -Note that the `user.yaml` example below is minimal, as the goal is only to describe its structure. For a working `user.yaml` file that contains everything needed to get started, refer to the [base user.yaml](base_user.yaml) instead. +Note that the `user.yaml` example below is minimal, as the goal is only to describe its structure. For a working `user.yaml` file that contains everything needed to get started, refer to the [base user.yaml](../base_user.yaml) instead. ``` authz: diff --git a/docs/usersync.md b/docs/misc/usersync.md similarity index 99% rename from docs/usersync.md rename to docs/misc/usersync.md index ef896c37b..8c0a5d79a 100644 --- a/docs/usersync.md +++ b/docs/misc/usersync.md @@ -6,7 +6,7 @@ Usersync is a script that parses user access information from multiple sources ( ## Usersync flow -![Usersync Flow](images/usersync.png) +![Usersync Flow](../images/usersync.png) > The access from the user.yaml file and the dbGaP authorization files is combined (see example below), but the user.yaml file overrides the user information (such as email) obtained from the dbGaP authorization files. From 00eee5629101aae0c4fe88f5724b0fe51b9a922e Mon Sep 17 00:00:00 2001 From: Albert Snow Date: Mon, 22 Jul 2024 12:45:57 -0500 Subject: [PATCH 3/9] moving readme content to folder --- docs/introduction/default_expiration_times.md | 19 +++ docs/introduction/fence_create.md | 126 ++++++++++++++ docs/introduction/oidc_and_oauth2.md | 74 +++++++++ docs/introduction/setup.md | 157 ++++++++++++++++++ docs/introduction/terminology.md | 51 ++++++ docs/introduction/token_management.md | 113 +++++++++++++ 6 files changed, 540 insertions(+) create mode 100644 docs/introduction/default_expiration_times.md create mode 100644 docs/introduction/fence_create.md create mode 100644 docs/introduction/oidc_and_oauth2.md create mode 100644 docs/introduction/setup.md create mode 100644 docs/introduction/terminology.md create mode 100644 docs/introduction/token_management.md diff --git a/docs/introduction/default_expiration_times.md b/docs/introduction/default_expiration_times.md new file mode 100644 index 000000000..a81528a1f --- /dev/null +++ b/docs/introduction/default_expiration_times.md @@ -0,0 +1,19 @@ +## Default Expiration Times in Fence + +Table contains various artifacts in fence that have temporary lifetimes and their default values. + +> NOTE: "SA" in the below table stands for Service Account + +| Name | Lifetime | Extendable? | Maximum Lifetime | Details +|--------------------------------------|--------------|-------------|-----------------------|------------------------------------------------------------------------------------------| +| Access Token | 20 minutes | TRUE | Life of Refresh Token | | +| Refresh Token | 30 days | FALSE | N/A | | +| User's SA Account Access | 7 days | TRUE | N/A | Access to data (e.g. length it stays in the proxy group). Can optionally provide an expiration less than 7 days | +| User's Google Account Access | 1 day | TRUE | N/A | After AuthN, how long we associate a Google email with the given user. Can optionally provide an expiration less than 1 day | +| User's Google Account Linkage | Indefinite | N/A | N/A | Can optionally provide an expiration less than 1 hour | +| Google Signed URL | Up to 1 hour | FALSE | N/A | Can optionally provide an expiration less than 1 hour | +| AWS Signed URL | Up to 1 hour | FALSE | N/A | Obtained by an oauth client through /credentials/google | +| Client SA (for User) Key | 10 days | FALSE | N/A | Obtained by the user themselves for temp access. Can optionally provide an expiration less than 10 days | +| User Primary SA Key | 10 days | FALSE | N/A | Used for Google URL signing | +| User Primary SA Key for URL Signing | 30 days | FALSE | N/A | | +| Sliding Session Window | 15 minutes | TRUE | 8 hours | access_token cookies get generated automatically when expired if session is still active | diff --git a/docs/introduction/fence_create.md b/docs/introduction/fence_create.md new file mode 100644 index 000000000..398544f98 --- /dev/null +++ b/docs/introduction/fence_create.md @@ -0,0 +1,126 @@ +## fence-create: Automating common tasks with a command line interface + +fence-create is a command line utility that is bundled with fence and allows you to automate some commons tasks within fence. For the latest and greatest run the command `fence-create --help`. + +WARNING: fence-create directly modifies the database in some cases and may circumvent security checks (most of these utilities are used for testing). BE CAREFUL when you're running these commands and make sure you know what they're doing. + + +### Register Internal Oauth Client + +As a Gen3 commons administrator, if you want to create an oauth client that skips user consent step, use the following command: + +```bash +fence-create client-create --client CLIENT_NAME --urls OAUTH_REDIRECT_URL --username USERNAME --auto-approve (--expires-in 30) +``` + +The optional `--expires-in` parameter allows specifying the number of days until this client expires. + +### Register an Implicit Oauth Client + +As a Gen3 commons administrator, if you want to create an implicit oauth client for a webapp: + +```bash +fence-create client-create --client fancywebappname --urls 'https://betawebapp.example/fence +https://webapp.example/fence' --public --username fancyapp --grant-types authorization_code refresh_token implicit +``` + +If there are more than one URL to add, use space to delimit them like this: + +```bash +fence-create client-create --urls 'https://url1/' 'https://url2/' --client ... +``` + +To specify allowed scopes, use the `allowed-scopes` argument: +```bash +fence-create client-create ... --allowed-scopes openid user data +``` + +### Register an Oauth Client for a Client Credentials flow + +The OAuth2 Client Credentials flow is used for machine-to-machine communication and scenarios in which typical authentication schemes like username + password do not make sense. The system authenticates and authorizes the app rather than a user. See the [OAuth2 specification](https://www.rfc-editor.org/rfc/rfc6749#section-4.4) for more details. + +As a Gen3 commons administrator, if you want to create an OAuth client for a client credentials flow: + +```bash +fence-create client-create --client CLIENT_NAME --grant-types client_credentials (--expires-in 30) +``` + +This command will return a client ID and client secret, which you can then use to obtain an access token: + +```bash +curl --request POST https://FENCE_URL/oauth2/token?grant_type=client_credentials -d scope="openid user" --user CLIENT_ID:CLIENT_SECRET +``` + +The optional `--expires-in` parameter allows specifying the number of *days* until this client expires. The recommendation is to rotate credentials with the `client_credentials` grant at least once a year (see [Rotate client credentials](#rotate-client-credentials) section). + +NOTE: In Gen3, you can grant specific access to a client the same way you would to a user. See the [user.yaml guide](https://github.com/uc-cdis/fence/blob/master/docs/user.yaml_guide.md) for more details. + +NOTE: Client credentials tokens are not linked to a user (the claims contain no `sub` or `context.user.name` like other tokens). Some Gen3 endpoints that assume the token is linked to a user, or whose logic require there being a user, do not support them. For an example of how to adapt an endpoint to support client credentials tokens, see [here](https://github.com/uc-cdis/requestor/commit/a5078fae27fa258ac78045cf2bb89cb2104f53cf). For an example of how to explicitly reject client credentials tokens, see [here](https://github.com/uc-cdis/requestor/commit/0f4974c25343d2185c7cdb48dcdeb58f97800672). + +### Modify OAuth Client + +```bash +fence-create client-modify --client CLIENT_NAME --urls http://localhost/api/v0/oauth2/authorize +``` + +That command should output any modifications to the client. Similarly, multiple URLs are +allowed here too. + +Add `--append` argument to add new callback urls or allowed scopes to existing client (instead of replacing them) using `--append --urls` or `--append --allowed-scopes` +```bash +fence-create client-modify --client CLIENT_NAME --urls http://localhost/api/v0/new/oauth2/authorize --append (--expires-in 30) +``` + +### Rotate client credentials + +Use the `client-rotate` command to receive a new set of credentials (client ID and secret) for a client. The old credentials are NOT deactivated and must be deleted or expired separately (see [Delete Expired OAuth Clients](#delete-expired-oauth-clients) section). This allows for a rotation without downtime. + +```bash +fence-create client-rotate --client CLIENT_NAME (--expires-in 30) +``` + +Note that the `usersync` job must be run after rotating the credentials so that the new client ID is granted the same access as the old one. + +### Delete OAuth Client + +```bash +fence-create client-delete --client CLIENT_NAME +``` +That command should output the result of the deletion attempt. + +### Delete Expired OAuth Clients + +```bash +fence-create client-delete-expired +``` + +To post a warning in Slack about any clients that expired or are about to expire: + +```bash +fence-create client-delete-expired --slack-webhook --warning-days +``` + + +### List OAuth Clients + +```bash +fence-create client-list +``` +That command should output the full records for any registered OAuth clients. + +### Set up for External Buckets on Google + +```bash +fence-create link-external-bucket --bucket-name demo-bucket +fence-create link-bucket-to-project --bucket_id demo-bucket --bucket_provider google --project_auth_id test-project +``` + +The link-external-bucket returns an email for a Google group which needs to be added to access to the bucket `demo-bucket`. + +### Notify users who are blocking service account registration + +```bash +fence-create notify-problem-users --emails ex1@gmail.com ex2@gmail.com --auth_ids test --google_project_id test-google +``` + +`notify-problem-users` emails users in the provided list (can be fence user email or linked google email) who do not have access to any of the auth_ids provided. Also accepts a `check_linking` flag to check that each user has linked their google account. diff --git a/docs/introduction/oidc_and_oauth2.md b/docs/introduction/oidc_and_oauth2.md new file mode 100644 index 000000000..8cc1e6a7a --- /dev/null +++ b/docs/introduction/oidc_and_oauth2.md @@ -0,0 +1,74 @@ +## OIDC & OAuth2 + +Fence acts as a central broker that supports multiple IdPs. +At the same time, it acts as an IdP itself. +In that sense, `fence` is both an `RP` and an `OP`. + +### Fence as RP + +Example: + +- Google IAM is the OpenID Provider (OP) +- Fence is the Relying Party (RP) +- Google Calendar API is the resource provider + +### Fence as OP + +- Fence is the OpenID Provider (OP) +- A third-party application is the Relying Party (RP) +- Gen3 microservices (e.g. [`sheepdog`](https://github.com/uc-cdis/sheepdog)) are resource providers + +### Example Flows + +Note that the `3rd Party App` acts as the `RP` in these examples. + +[//]: # (See /docs folder for README on how to regenerate these sequence diagrams) + +#### Flow: Client Registration + +![Client Registration](docs/images/seq_diagrams/client_registration.png) + +#### Flow: OpenID Connect + +In the following flow, Fence and the IdP together constitute an `OP`. +Fence, by itself, acts as an OAuth 2.0 Auth Server; the IdP enables the additional implementation of OIDC (by providing AuthN). From an OIDC viewpoint, therefore, Fence and the IdP can be abstracted into one `OP`. + +![OIDC Flow](docs/images/seq_diagrams/openid_connect_flow.png) + +If the third-party application doesn't need to use any Gen3 resources (and just +wants to authenticate the user), they can just get +needed information in the `ID token` after the handshake is finished . + +#### Flow: Using Tokens for Access + +If a third-party application wants to use Gen3 resources like +`fence`/`sheepdog`/`peregrine`, they call those services with an `Access Token` +passed in an `Authorization` header. + +In the following flow, `3rd Party App` is the `RP`; `Protected Endpoint` is an endpoint of a Gen3 Resource (the `microservice`), and both of these are part of a `resource server`; and `Fence` is the `OP`. Here, importantly, `Fence` may be interfacing with another IdP _or_ with another `Fence` instance in order to implement the OIDC layer. Either way, note that the `Fence` blob in this diagram actually abstracts Fence in concert with some IdP, which may or may not also be (a different instance of) Fence. + +![Using Access Token](docs/images/seq_diagrams/token_use_for_access.png) + +#### Flow: Refresh Token Use + +![Using Refresh Token](docs/images/seq_diagrams/refresh_token_use.png) + +#### Flow: Refresh Token Use (Token is Expired) + +![Using Expired Refresh Token](docs/images/seq_diagrams/refresh_token_use_expired.png) + +#### Flow: Multi-Tenant Fence + +The following diagram illustrates the case in which one fence instance +uses another fence instance as its identity provider. + +A use case for this is when we setup a fence instance that uses NIH login as the IdP. Here, we go through a detailed approval process in NIH. Therefore we would like to do it only once for a single lead Fence instance, and then allow other fence instances to simply redirect to use the lead Fence as an IdP for logging in via NIH. + +In the following flow, `Fence (Client Instance)` is an OP relative to `OAuth Client`, but an RP relative to `Fence (IDP)`. + +![Multi-Tenant Flow](docs/images/seq_diagrams/multi-tenant_flow.png) + +#### Notes + +See the [OIDC specification](http://openid.net/specs/openid-connect-core-1_0.html) for more details. +Additionally, see the [OAuth2 specification](https://tools.ietf.org/html/rfc6749). diff --git a/docs/introduction/setup.md b/docs/introduction/setup.md new file mode 100644 index 000000000..5dcafc37b --- /dev/null +++ b/docs/introduction/setup.md @@ -0,0 +1,157 @@ + +## Setup + +### Install Requirements and Fence + +Install [Poetry](https://python-poetry.org/docs/#installation). + +```bash +# Install Fence and dependencies +poetry install +``` + +### Create Configuration File + +Fence requires a configuration file to run. We have a command line +utility to help you create one based on a default configuration. + +The configuration file itself will live outside of this repo (to +prevent accidentally checking in sensitive information like database passwords). + +To create a new configuration file from the default configuration: + +```bash +python cfg_help.py create +``` + +This file will be placed in one of the default search directories for Fence. + +To get the exact path where the new configuration file was created, use: + +```bash +python cfg_help.py get +``` + +The file should have detailed information about each of the configuration +variables. **Remember to fill out the new configuration file!** + +Once you have done so, you can run `alembic upgrade head` to generate the tables needed +to run fence. + +#### Other Configuration Notes + +* Fence will look for configuration files from a list of search directories ( +which are currently defined in `fence/settings.py`.) +* For more configuration options (such as having multiple different config +files for development), see the `cfg_help.py` file. + +### Set Up Databases + +The tests clear out the database every time they are run. If you want +to keep a persistent database for manual testing and general local usage, +create a second test database with a different name: + +> NOTE: Requires a minimum of Postgres v9.4 (because of `JSONB` types used) + +```bash +# Create test database(s). +# This one is for automated tests, which clear the database after running; +# `tests/test_settings.py` should have `fence_test_tmp` in the `DB` variable. +psql -U test postgres -c 'create database fence_test_tmp' +userdatamodel-init --db fence_test_tmp +# This one is for manual testing/general local usage; Your config +# should have `fence_test` in the `DB` variable. +psql -U test postgres -c 'create database fence_test' +userdatamodel-init --db fence_test --username test --password test +``` + +### Keypair Configuration + +Fence uses RSA keypairs to sign and allow verification of JWTs that it issues. +When the application is initialized, Fence loads in keypair files from the +`keys` directory. To store keypair files, use the following procedure: + - Create a subdirectory in the `fence/keys` directory, named with a + unique identifier, preferably a timestamp in ISO 8601 format of when + the keys are created. The name of the directory is used for the `kid` + (key ID) for those keys; the default (assuming the directory is named + with an ISO timestamp) looks like this: + + fence_key_2018-05-01T14:00:00Z + + - Generate a private and public keypair following the RSA 256 algorithm + and store those in that directory. The key files must be named + `jwt_public_key.pem` and `jwt_private_key.pem`. + +To generate a keypair using `openssl`: +```bash +# Generate the private key. +openssl genpkey -algorithm RSA -out jwt_private_key.pem -pkeyopt rsa_keygen_bits:2048 + +# Generate the public key. +openssl rsa -pubout -in jwt_private_key.pem -out jwt_public_key.pem + +# Depending on the `openssl` distribution, you may find these work instead: +# +# openssl rsa -out private_key.pem 2048 +# openssl rsa -in private_key.pem -pubout -out public_key.pem +``` +It's not a bad idea to confirm that the files actually say `RSA PRIVATE KEY` +and `PUBLIC KEY` (and in fact Fence will require that the private key files it +uses actually say "PRIVATE KEY" and that the public keys do not). + +Files containing public/private keys should have this format (the format used +by `openssl` for generating RSA keys): +``` +-----BEGIN PUBLIC KEY----- +... [key is here] ... +-----END PUBLIC KEY----- +``` +If a key is not in this format, then `PyJWT` will raise errors about not being +able to read the key. + +Fence will use the first keypair in the list to sign the tokens it issues +through OAuth. + + +### Create User Access File + +You can setup user access via admin fence script providing a user yaml file +Example user yaml: +``` +cloud_providers: {} +groups: {} +users: + userA@gmail.com: + projects: + - auth_id: project_a + privilege: [read, update, create, delete] + - auth_id: project_b + privilege: [read] + userB@gmail.com: + projects: + - auth_id: project_b + privilege: [read] +``` +Example sync command: + +```bash +fence-create sync --yaml user.yaml +``` + +### Register OAuth Client + +When you want to build an application that uses Gen3 resources on behalf of a user, you should register an OAuth client for this app. +Fence right now exposes client registration via admin CLI, because the Oauth2 client for a Gen3 commons needs approval from the sponsor of the commons. If you are an external developer, you should submit a support ticket. + +As a Gen3 commons administrator, you can run following command for an approved client: +```bash +fence-create client-create --client CLIENT_NAME --urls OAUTH_REDIRECT_URL --username USERNAME +``` +This command should output a tuple of `(client_id, client_secret)` which must be +saved by the OAuth client to use with +`fence`. + +## Quickstart with Helm + +You can now deploy individual services via Helm! +Please refer to the Helm quickstart guide HERE (https://github.com/uc-cdis/fence/blob/master/docs/quickstart_helm.md) diff --git a/docs/introduction/terminology.md b/docs/introduction/terminology.md new file mode 100644 index 000000000..8e3ffd812 --- /dev/null +++ b/docs/introduction/terminology.md @@ -0,0 +1,51 @@ +## Terminologies + +### AuthN + +Authentication - establishes "who you are" with the application through communication with an [Identity Provider](#IdP). + +### AuthZ + +Authorization - establishes "what you can do" and "which resources you have access to" within the application. + +### IdP + +Identity Provider - the service that lets a user login and provides the identity of the user to downstream services. Examples: Google login, University login, NIH Login. + +### Auth broker + +An interface which enables a user to authenticate using any of multiple IdPs. + +### OAuth2 + +A widely used AuthZ protocol for delegating access to an application to use resources on behalf of a user. + +https://tools.ietf.org/html/rfc6749 + +https://oauth.net/2/ + +#### Client + +OAuth 2.0 Client - An application which makes requests for protected resources (on a resource server) on behalf of a resource owner (end-user) and with the resource owner's authorization. + +#### Auth Server + +OAuth 2.0 Authorization Server - A server which issues access tokens to the client after successfully authenticating the resource owner and obtaining authorization. + +#### Access Token + +A string, issued by the auth server to the client, representing authorization credentials used to access protected resources (on a resource server). + +### OIDC + +OpenID Connect - an extension of OAuth2 which provides an AuthN layer on top of the OAuth 2.0 AuthZ layer. It introduced a new type of token, the id token, that is specifically designed to be consumed by clients to get the identity information of the user. + +http://openid.net/specs/openid-connect-core-1_0.html + +#### OP + +OpenID Provider - an OAuth 2.0 Authentication Server which also implements OpenID Connect. + +#### RP + +Relying Party - an OAuth 2.0 Client which uses (requests) OpenID Connect. diff --git a/docs/introduction/token_management.md b/docs/introduction/token_management.md new file mode 100644 index 000000000..7d59e1d2e --- /dev/null +++ b/docs/introduction/token_management.md @@ -0,0 +1,113 @@ + +## Token management + +Fence utilizes [OpenID Connect](#OIDC) to generate tokens +for clients. It can also provide tokens directly to a user. + +Clients and users may then use those tokens with other +Gen3 Data Commons services to access protected endpoints that require specific permissions. + +We use JSON Web Tokens (JWTs) as the format for all tokens of the following types: + +- OIDC ID token: this token is used by the OIDC client to get a user's identity from the token content +- OIDC access token: this token can be sent to Gen3 services via bearer header and get protected resources. +- OIDC refresh token: this token can be sent to fence to request a new access / id token. + + + +### JWT Information + +#### Example ID Token + +``` +{ + "sub": "7", + "azp": "test-client", + "pur": "id", + "aud": [ + "openid", + "user", + "test-client" + ], + "context": { + "user": { + "is_admin": false, + "name": "test", + "projects": { + "phs000178": [ + "read", + "update", + "create", + "delete", + "read-storage" + ] + }, + "google": { + "linked_google_account": "somebody@example.com" + } + } + }, + "iss": "https://commons.org", + "jti": "3ae2910b-0294-43dc-af2a-03fd60082aef", + "exp": 1516983302, + "iat": 1516982102, + "auth_time": 1516982102 +} +``` + +#### Example Access Token + +``` +{ + "sub": "7", + "azp": "test-client", + "pur": "access", + "aud": [ + "openid", + "user", + "test-client" + ], + "context": { + "user": { + "is_admin": false, + "name": "test", + "projects": { + "phs000178": [ + "read", + "update", + "create", + "delete", + "read-storage" + ] + }, + "google": { + "proxy_group": "abcdefgh123456", + "linked_google_account": "somebody@example.com" + } + } + }, + "iss": "https://commons.org", + "jti": "2e6ade06-5afb-4ce7-9ab5-e206225ce291", + "exp": 1516983302, + "iat": 1516982102 +} +``` + +#### Example Refresh Token + +``` +{ + "sub": "7", + "azp": "test-client", + "pur": "refresh", + "aud": [ + "openid", + "user", + "test-client" + ], + "iss": "https://commons.org", + "jti": "c72e5573-39fa-4391-a445-191e370b7cc5", + "exp": 1517010902, + "iat": 1516982102 +} +``` \ No newline at end of file From f102fab0f212db1e58540de764cae1c57705d08f Mon Sep 17 00:00:00 2001 From: Albert Snow Date: Mon, 22 Jul 2024 12:56:57 -0500 Subject: [PATCH 4/9] cleaning up references --- README.md | 22 ++++++++-------- .../dbgap_info.md | 0 .../fence-create-deprecated.md | 0 .../fence_multifactor_authentication_guide.md | 0 .../fence_shibboleth.md | 0 .../ga4gh_passports.md | 0 .../google_architecture.md | 0 .../local_multi_fence.md | 0 .../quickstart_helm.md | 0 .../register.md | 0 .../user.yaml_guide.md | 0 .../usersync.md | 0 docs/introduction/default_expiration_times.md | 26 +++++++++---------- docs/introduction/oidc_and_oauth2.md | 12 ++++----- docs/introduction/token_management.md | 2 +- 15 files changed, 31 insertions(+), 31 deletions(-) rename docs/{misc => additional_documentation}/dbgap_info.md (100%) rename docs/{misc => additional_documentation}/fence-create-deprecated.md (100%) rename docs/{misc => additional_documentation}/fence_multifactor_authentication_guide.md (100%) rename docs/{misc => additional_documentation}/fence_shibboleth.md (100%) rename docs/{misc => additional_documentation}/ga4gh_passports.md (100%) rename docs/{misc => additional_documentation}/google_architecture.md (100%) rename docs/{misc => additional_documentation}/local_multi_fence.md (100%) rename docs/{misc => additional_documentation}/quickstart_helm.md (100%) rename docs/{misc => additional_documentation}/register.md (100%) rename docs/{misc => additional_documentation}/user.yaml_guide.md (100%) rename docs/{misc => additional_documentation}/usersync.md (100%) diff --git a/README.md b/README.md index 9b18c29dc..b900de2ec 100644 --- a/README.md +++ b/README.md @@ -9,9 +9,9 @@ only trusted entities to enter. Fence is a core service of the Gen3 stack that has multiple capabilities: -1. Act as an [auth broker](#auth-broker) to integrate with one or more [IdPs](#IdP) and provide downstream authentication and authorization for Gen3 services. -2. [Manage tokens](#token-management). -3. Act as an [OIDC provider](#oidc--oauth2) to support external applications to use Gen3 services. +1. Act as an [auth broker](./docs/introduction/terminology.md#auth-broker) to integrate with one or more [IdPs](#IdP) and provide downstream authentication and authorization for Gen3 services. +2. [Manage tokens](./docs/introduction/token_management.md). +3. Act as an [OIDC provider](./docs/introduction/oidc_and_oauth2.md) to support external applications to use Gen3 services. 4. [Issue short lived, cloud native credentials to access data in various cloud storage services](#accessing-data) @@ -20,12 +20,12 @@ Fence is a core service of the Gen3 stack that has multiple capabilities: 1. [API Documentation](#API-documentation) 1. [Terminologies](./docs/introduction/terminology.md) 1. [Identity Providers](#identity-providers) -1. [OIDC & OAuth2](#oidc--oauth2) +1. [OIDC & OAuth2](./docs/introduction/oidc_and_oauth2.md) 1. [Accessing Data](#accessing-data) -1. [Setup](#setup) -1. [Token management](#token-management) -1. [fence-create](#fence-create-automating-common-tasks-with-a-command-line-interface) -1. [Default expiration times](#default-expiration-times-in-fence) +1. [Setup](./docs/introduction/setup.md) +1. [Token management](./docs/introduction/token_management.md) +1. [fence-create](./docs/introduction/fence_create.md) +1. [Default expiration times](./docs/introduction/default_expiration_times.md) ## API Documentation @@ -42,7 +42,7 @@ Fence can be configured to support different Identity Providers (IdPs) for AuthN At the moment, supported IDPs include: - Google -- [Shibboleth](docs/misc/fence_shibboleth.md) +- [Shibboleth](docs/additional_documentation/fence_shibboleth.md) - NIH iTrust - InCommon - eduGAIN @@ -58,7 +58,7 @@ At the moment, supported IDPs include: Currently fence works with another Gen3 service named [arborist](https://github.com/uc-cdis/arborist) to implement attribute-based access control for commons users. The YAML file of access control information (see -[#create-user-access-file](#create-user-access-file)) contains a section `authz` which are data sent to +[#create-user-access-file](./docs/introduction/setup.md#create-user-access-file)) contains a section `authz` which are data sent to arborist in order to set up the access control model. ## Accessing Data @@ -88,6 +88,6 @@ Fence has the ability to request a specific file by its GUID (globally unique id Whereas pre-signed URL is a cloud agnostic solution, services and tools on Google Cloud Platform prefer to use Google's concept of a "Service Account". Because of that, Fence provides a few more methods to access data in Google. -See [Fence and Google](docs/misc/google_architecture.md) for more details on data access methods specific to Google. +See [Fence and Google](docs/additional_documentation/google_architecture.md) for more details on data access methods specific to Google. diff --git a/docs/misc/dbgap_info.md b/docs/additional_documentation/dbgap_info.md similarity index 100% rename from docs/misc/dbgap_info.md rename to docs/additional_documentation/dbgap_info.md diff --git a/docs/misc/fence-create-deprecated.md b/docs/additional_documentation/fence-create-deprecated.md similarity index 100% rename from docs/misc/fence-create-deprecated.md rename to docs/additional_documentation/fence-create-deprecated.md diff --git a/docs/misc/fence_multifactor_authentication_guide.md b/docs/additional_documentation/fence_multifactor_authentication_guide.md similarity index 100% rename from docs/misc/fence_multifactor_authentication_guide.md rename to docs/additional_documentation/fence_multifactor_authentication_guide.md diff --git a/docs/misc/fence_shibboleth.md b/docs/additional_documentation/fence_shibboleth.md similarity index 100% rename from docs/misc/fence_shibboleth.md rename to docs/additional_documentation/fence_shibboleth.md diff --git a/docs/misc/ga4gh_passports.md b/docs/additional_documentation/ga4gh_passports.md similarity index 100% rename from docs/misc/ga4gh_passports.md rename to docs/additional_documentation/ga4gh_passports.md diff --git a/docs/misc/google_architecture.md b/docs/additional_documentation/google_architecture.md similarity index 100% rename from docs/misc/google_architecture.md rename to docs/additional_documentation/google_architecture.md diff --git a/docs/misc/local_multi_fence.md b/docs/additional_documentation/local_multi_fence.md similarity index 100% rename from docs/misc/local_multi_fence.md rename to docs/additional_documentation/local_multi_fence.md diff --git a/docs/misc/quickstart_helm.md b/docs/additional_documentation/quickstart_helm.md similarity index 100% rename from docs/misc/quickstart_helm.md rename to docs/additional_documentation/quickstart_helm.md diff --git a/docs/misc/register.md b/docs/additional_documentation/register.md similarity index 100% rename from docs/misc/register.md rename to docs/additional_documentation/register.md diff --git a/docs/misc/user.yaml_guide.md b/docs/additional_documentation/user.yaml_guide.md similarity index 100% rename from docs/misc/user.yaml_guide.md rename to docs/additional_documentation/user.yaml_guide.md diff --git a/docs/misc/usersync.md b/docs/additional_documentation/usersync.md similarity index 100% rename from docs/misc/usersync.md rename to docs/additional_documentation/usersync.md diff --git a/docs/introduction/default_expiration_times.md b/docs/introduction/default_expiration_times.md index a81528a1f..9b0432270 100644 --- a/docs/introduction/default_expiration_times.md +++ b/docs/introduction/default_expiration_times.md @@ -4,16 +4,16 @@ Table contains various artifacts in fence that have temporary lifetimes and thei > NOTE: "SA" in the below table stands for Service Account -| Name | Lifetime | Extendable? | Maximum Lifetime | Details -|--------------------------------------|--------------|-------------|-----------------------|------------------------------------------------------------------------------------------| -| Access Token | 20 minutes | TRUE | Life of Refresh Token | | -| Refresh Token | 30 days | FALSE | N/A | | -| User's SA Account Access | 7 days | TRUE | N/A | Access to data (e.g. length it stays in the proxy group). Can optionally provide an expiration less than 7 days | -| User's Google Account Access | 1 day | TRUE | N/A | After AuthN, how long we associate a Google email with the given user. Can optionally provide an expiration less than 1 day | -| User's Google Account Linkage | Indefinite | N/A | N/A | Can optionally provide an expiration less than 1 hour | -| Google Signed URL | Up to 1 hour | FALSE | N/A | Can optionally provide an expiration less than 1 hour | -| AWS Signed URL | Up to 1 hour | FALSE | N/A | Obtained by an oauth client through /credentials/google | -| Client SA (for User) Key | 10 days | FALSE | N/A | Obtained by the user themselves for temp access. Can optionally provide an expiration less than 10 days | -| User Primary SA Key | 10 days | FALSE | N/A | Used for Google URL signing | -| User Primary SA Key for URL Signing | 30 days | FALSE | N/A | | -| Sliding Session Window | 15 minutes | TRUE | 8 hours | access_token cookies get generated automatically when expired if session is still active | +| Name | Lifetime | Extendable? | Maximum Lifetime | Details | +|-------------------------------------|--------------|-------------|-----------------------|-----------------------------------------------------------------------------------------------------------------------------| +| Access Token | 20 minutes | TRUE | Life of Refresh Token | | +| Refresh Token | 30 days | FALSE | N/A | | +| User's SA Account Access | 7 days | TRUE | N/A | Access to data (e.g. length it stays in the proxy group). Can optionally provide an expiration less than 7 days | +| User's Google Account Access | 1 day | TRUE | N/A | After AuthN, how long we associate a Google email with the given user. Can optionally provide an expiration less than 1 day | +| User's Google Account Linkage | Indefinite | N/A | N/A | Can optionally provide an expiration less than 1 hour | +| Google Signed URL | Up to 1 hour | FALSE | N/A | Can optionally provide an expiration less than 1 hour | +| AWS Signed URL | Up to 1 hour | FALSE | N/A | Obtained by an oauth client through /credentials/google | +| Client SA (for User) Key | 10 days | FALSE | N/A | Obtained by the user themselves for temp access. Can optionally provide an expiration less than 10 days | +| User Primary SA Key | 10 days | FALSE | N/A | Used for Google URL signing | +| User Primary SA Key for URL Signing | 30 days | FALSE | N/A | | +| Sliding Session Window | 15 minutes | TRUE | 8 hours | access_token cookies get generated automatically when expired if session is still active | diff --git a/docs/introduction/oidc_and_oauth2.md b/docs/introduction/oidc_and_oauth2.md index 8cc1e6a7a..b979a32e2 100644 --- a/docs/introduction/oidc_and_oauth2.md +++ b/docs/introduction/oidc_and_oauth2.md @@ -26,14 +26,14 @@ Note that the `3rd Party App` acts as the `RP` in these examples. #### Flow: Client Registration -![Client Registration](docs/images/seq_diagrams/client_registration.png) +![Client Registration](../images/seq_diagrams/client_registration.png) #### Flow: OpenID Connect In the following flow, Fence and the IdP together constitute an `OP`. Fence, by itself, acts as an OAuth 2.0 Auth Server; the IdP enables the additional implementation of OIDC (by providing AuthN). From an OIDC viewpoint, therefore, Fence and the IdP can be abstracted into one `OP`. -![OIDC Flow](docs/images/seq_diagrams/openid_connect_flow.png) +![OIDC Flow](../images/seq_diagrams/openid_connect_flow.png) If the third-party application doesn't need to use any Gen3 resources (and just wants to authenticate the user), they can just get @@ -47,15 +47,15 @@ passed in an `Authorization` header. In the following flow, `3rd Party App` is the `RP`; `Protected Endpoint` is an endpoint of a Gen3 Resource (the `microservice`), and both of these are part of a `resource server`; and `Fence` is the `OP`. Here, importantly, `Fence` may be interfacing with another IdP _or_ with another `Fence` instance in order to implement the OIDC layer. Either way, note that the `Fence` blob in this diagram actually abstracts Fence in concert with some IdP, which may or may not also be (a different instance of) Fence. -![Using Access Token](docs/images/seq_diagrams/token_use_for_access.png) +![Using Access Token](../images/seq_diagrams/token_use_for_access.png) #### Flow: Refresh Token Use -![Using Refresh Token](docs/images/seq_diagrams/refresh_token_use.png) +![Using Refresh Token](../images/seq_diagrams/refresh_token_use.png) #### Flow: Refresh Token Use (Token is Expired) -![Using Expired Refresh Token](docs/images/seq_diagrams/refresh_token_use_expired.png) +![Using Expired Refresh Token](../images/seq_diagrams/refresh_token_use_expired.png) #### Flow: Multi-Tenant Fence @@ -66,7 +66,7 @@ A use case for this is when we setup a fence instance that uses NIH login as the In the following flow, `Fence (Client Instance)` is an OP relative to `OAuth Client`, but an RP relative to `Fence (IDP)`. -![Multi-Tenant Flow](docs/images/seq_diagrams/multi-tenant_flow.png) +![Multi-Tenant Flow](../images/seq_diagrams/multi-tenant_flow.png) #### Notes diff --git a/docs/introduction/token_management.md b/docs/introduction/token_management.md index 7d59e1d2e..01458716a 100644 --- a/docs/introduction/token_management.md +++ b/docs/introduction/token_management.md @@ -1,7 +1,7 @@ ## Token management -Fence utilizes [OpenID Connect](#OIDC) to generate tokens +Fence utilizes [OpenID Connect](./terminology.md#oidc) to generate tokens for clients. It can also provide tokens directly to a user. Clients and users may then use those tokens with other From 21744ed20f341bb216b93243b1fa6336098ec2a1 Mon Sep 17 00:00:00 2001 From: Albert Snow Date: Mon, 22 Jul 2024 13:08:10 -0500 Subject: [PATCH 5/9] fixing broken link --- docs/azure/azure_architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/azure/azure_architecture.md b/docs/azure/azure_architecture.md index 955bde7ac..a7bf1d170 100755 --- a/docs/azure/azure_architecture.md +++ b/docs/azure/azure_architecture.md @@ -122,7 +122,7 @@ Also note that there's alternatives that could be considered for [future develop It's helpful to understand some of the [Arborist terms and definitions](https://github.com/uc-cdis/arborist#terminology-and-definitions), which covers **action**, **permission**, **role**, **resource**, **policy**, and **group**. -Further, it's helpful to understand the Arborist options for [configuring access](https://github.com/uc-cdis/arborist#configuring-access). You can see an example of granting **users** and **groups** access and more details in the [user.yaml guide](../misc/user.yaml_guide.md). +Further, it's helpful to understand the Arborist options for [configuring access](https://github.com/uc-cdis/arborist#configuring-access). You can see an example of granting **users** and **groups** access and more details in the [user.yaml guide](../additional_documentation/user.yaml_guide.md). At a high level, this setup involves a couple steps: From a2d055bd67e52cca2e2d200855f59a8f6a77f93f Mon Sep 17 00:00:00 2001 From: Albert Snow Date: Tue, 23 Jul 2024 10:00:05 -0500 Subject: [PATCH 6/9] implementing alex's suggestions --- README.md | 140 ++++++++++++------ .../additional_documentation/authorization.md | 8 + docs/additional_documentation/data_access.md | 22 +++ .../default_expiration_times.md | 0 .../fence_create.md | 0 .../setup.md | 0 .../terminology.md | 0 .../token_management.md | 2 +- docs/introduction/oidc_and_oauth2.md | 74 --------- 9 files changed, 126 insertions(+), 120 deletions(-) create mode 100644 docs/additional_documentation/authorization.md create mode 100644 docs/additional_documentation/data_access.md rename docs/{introduction => additional_documentation}/default_expiration_times.md (100%) rename docs/{introduction => additional_documentation}/fence_create.md (100%) rename docs/{introduction => additional_documentation}/setup.md (100%) rename docs/{introduction => additional_documentation}/terminology.md (100%) rename docs/{introduction => additional_documentation}/token_management.md (96%) delete mode 100644 docs/introduction/oidc_and_oauth2.md diff --git a/README.md b/README.md index b900de2ec..84bd4d22f 100644 --- a/README.md +++ b/README.md @@ -9,35 +9,20 @@ only trusted entities to enter. Fence is a core service of the Gen3 stack that has multiple capabilities: -1. Act as an [auth broker](./docs/introduction/terminology.md#auth-broker) to integrate with one or more [IdPs](#IdP) and provide downstream authentication and authorization for Gen3 services. -2. [Manage tokens](./docs/introduction/token_management.md). -3. Act as an [OIDC provider](./docs/introduction/oidc_and_oauth2.md) to support external applications to use Gen3 services. -4. [Issue short lived, cloud native credentials to access data in various cloud storage services](#accessing-data) +1. Act as an [auth broker](docs/additional_documentation/terminology.md#auth-broker) to integrate with one +or more [IdPs](docs/additional_documentation/terminology.md#idp) and provide downstream authentication +and authorization for Gen3 services. +2. [Manage tokens](docs/additional_documentation/token_management.md). +3. Act as an [OIDC provider](README.md#oidc--oauth2) to support external +applications to use Gen3 services. +4. [Issue short lived, cloud native credentials to access data in various cloud storage services](docs/additional_documentation/data_access.md#accessing-data) -## Contents +## Overview -1. [API Documentation](#API-documentation) -1. [Terminologies](./docs/introduction/terminology.md) -1. [Identity Providers](#identity-providers) -1. [OIDC & OAuth2](./docs/introduction/oidc_and_oauth2.md) -1. [Accessing Data](#accessing-data) -1. [Setup](./docs/introduction/setup.md) -1. [Token management](./docs/introduction/token_management.md) -1. [fence-create](./docs/introduction/fence_create.md) -1. [Default expiration times](./docs/introduction/default_expiration_times.md) +### Identity Providers -## API Documentation - -[OpenAPI documentation available here.](http://petstore.swagger.io/?url=https://raw.githubusercontent.com/uc-cdis/fence/master/openapis/swagger.yaml) - -YAML file for the OpenAPI documentation is found in the `openapis` folder (in -the root directory); see the README in that folder for more details. - - -## Identity Providers - Fence can be configured to support different Identity Providers (IdPs) for AuthN. At the moment, supported IDPs include: @@ -53,41 +38,106 @@ At the moment, supported IDPs include: - ORCID - RAS -## Access Control / Authz +### ## API Documentation + +[OpenAPI documentation available here.](http://petstore.swagger.io/?url=https://raw.githubusercontent.com/uc-cdis/fence/master/openapis/swagger.yaml) + +YAML file for the OpenAPI documentation is found in the `openapis` folder (in +the root directory); see the README in that folder for more details. + + +## OIDC & OAuth2 + +Fence acts as a central broker that supports multiple IdPs. +At the same time, it acts as an IdP itself. +In that sense, `fence` is both an `RP` and an `OP`. + +### Fence as RP + +Example: + +- Google IAM is the OpenID Provider (OP) +- Fence is the Relying Party (RP) +- Google Calendar API is the resource provider + +### Fence as OP -Currently fence works with another Gen3 service named -[arborist](https://github.com/uc-cdis/arborist) to implement attribute-based access -control for commons users. The YAML file of access control information (see -[#create-user-access-file](./docs/introduction/setup.md#create-user-access-file)) contains a section `authz` which are data sent to -arborist in order to set up the access control model. +- Fence is the OpenID Provider (OP) +- A third-party application is the Relying Party (RP) +- Gen3 microservices (e.g. [`sheepdog`](https://github.com/uc-cdis/sheepdog)) are resource providers -## Accessing Data +### Example Flows -Fence has multiple options that provide a mechanism to access data. The access -to data can be moderated through authorization information in a User Access File. +Note that the `3rd Party App` acts as the `RP` in these examples. -Users can be provided specific `privilege`'s on `projects` in the User Access -File. A `project` is identified by a unique authorization identifier AKA `auth_id`. +[//]: # (See /docs folder for README on how to regenerate these sequence diagrams) -A `project` can be associated with various storage backends that store -object data for that given `project`. You can assign `read-storage` and `write-storage` -privileges to users who should have access to that stored object data. `read` and -`write` allow access to the data stored in a graph database. +#### Flow: Client Registration -Depending on the backend, Fence can be configured to provide users access to -the data in different ways. +![Client Registration](./docs/images/seq_diagrams/client_registration.png) +#### Flow: OpenID Connect -### Signed URLS +In the following flow, Fence and the IdP together constitute an `OP`. +Fence, by itself, acts as an OAuth 2.0 Auth Server; the IdP enables the additional implementation of OIDC (by providing AuthN). From an OIDC viewpoint, therefore, Fence and the IdP can be abstracted into one `OP`. -Temporary signed URLs are supported in all major commercial clouds. Signed URLs are the most 'cloud agnostic' way to allow users to access data located in different platforms. +![OIDC Flow](./docs/images/seq_diagrams/openid_connect_flow.png) + +If the third-party application doesn't need to use any Gen3 resources (and just +wants to authenticate the user), they can just get +needed information in the `ID token` after the handshake is finished . + +#### Flow: Using Tokens for Access + +If a third-party application wants to use Gen3 resources like +`fence`/`sheepdog`/`peregrine`, they call those services with an `Access Token` +passed in an `Authorization` header. + +In the following flow, `3rd Party App` is the `RP`; `Protected Endpoint` is an endpoint of a Gen3 Resource (the `microservice`), and both of these are part of a `resource server`; and `Fence` is the `OP`. Here, importantly, `Fence` may be interfacing with another IdP _or_ with another `Fence` instance in order to implement the OIDC layer. Either way, note that the `Fence` blob in this diagram actually abstracts Fence in concert with some IdP, which may or may not also be (a different instance of) Fence. + +![Using Access Token](./docs/images/seq_diagrams/token_use_for_access.png) + +#### Flow: Refresh Token Use + +![Using Refresh Token](./docs/images/seq_diagrams/refresh_token_use.png) + +#### Flow: Refresh Token Use (Token is Expired) + +![Using Expired Refresh Token](./docs/images/seq_diagrams/refresh_token_use_expired.png) + +#### Flow: Multi-Tenant Fence + +The following diagram illustrates the case in which one fence instance +uses another fence instance as its identity provider. + +A use case for this is when we setup a fence instance that uses NIH login as the IdP. Here, we go through a detailed approval process in NIH. Therefore we would like to do it only once for a single lead Fence instance, and then allow other fence instances to simply redirect to use the lead Fence as an IdP for logging in via NIH. + +In the following flow, `Fence (Client Instance)` is an OP relative to `OAuth Client`, but an RP relative to `Fence (IDP)`. + +![Multi-Tenant Flow](./docs/images/seq_diagrams/multi-tenant_flow.png) + +#### Notes + +See the [OIDC specification](http://openid.net/specs/openid-connect-core-1_0.html) for more details. +Additionally, see the [OAuth2 specification](https://tools.ietf.org/html/rfc6749). -Fence has the ability to request a specific file by its GUID (globally unique identifier) and retrieve a temporary signed URL for object data in AWS or GCP that will provide direct access to that object. ### Google Cloud Storage -Whereas pre-signed URL is a cloud agnostic solution, services and tools on Google Cloud Platform prefer to use Google's concept of a "Service Account". Because of that, Fence provides a few more methods to access data in Google. +Whereas pre-signed URL is a cloud-agnostic solution, services and tools on Google Cloud Platform prefer to use Google's concept of a "Service Account". Because of that, Fence provides a few more methods to access data in Google. See [Fence and Google](docs/additional_documentation/google_architecture.md) for more details on data access methods specific to Google. +## Setup + +See detailed explination [here](docs/additional_documentation/setup.md) + +## Additional documentation + +1. [Terminologies](docs/additional_documentation/terminology.md) +2. [Accessing Data](docs/additional_documentation/data_access.md#accessing-data) +3. [Setup](docs/additional_documentation/setup.md) +4. [Token management](docs/additional_documentation/token_management.md) +5. [fence-create](docs/additional_documentation/fence_create.md) +6. [Default expiration times](docs/additional_documentation/default_expiration_times.md) diff --git a/docs/additional_documentation/authorization.md b/docs/additional_documentation/authorization.md new file mode 100644 index 000000000..6fc31c7c1 --- /dev/null +++ b/docs/additional_documentation/authorization.md @@ -0,0 +1,8 @@ + +## Access Control / Authz + +Currently fence works with another Gen3 service named +[arborist](https://github.com/uc-cdis/arborist) to implement attribute-based access +control for commons users. The YAML file of access control information (see +[#create-user-access-file](setup.md#create-user-access-file)) contains a section `authz` which are data sent to +arborist in order to set up the access control model. diff --git a/docs/additional_documentation/data_access.md b/docs/additional_documentation/data_access.md new file mode 100644 index 000000000..3e5190fc9 --- /dev/null +++ b/docs/additional_documentation/data_access.md @@ -0,0 +1,22 @@ +## Accessing Data + +Fence has multiple options that provide a mechanism to access data. The access +to data can be moderated through authorization information in a User Access File. + +Users can be provided specific `privilege`'s on `projects` in the User Access +File. A `project` is identified by a unique authorization identifier AKA `auth_id`. + +A `project` can be associated with various storage backends that store +object data for that given `project`. You can assign `read-storage` and `write-storage` +privileges to users who should have access to that stored object data. `read` and +`write` allow access to the data stored in a graph database. + +Depending on the backend, Fence can be configured to provide users access to +the data in different ways. + + +### Signed URLS + +Temporary signed URLs are supported in all major commercial clouds. Signed URLs are the most 'cloud agnostic' way to allow users to access data located in different platforms. + +Fence has the ability to request a specific file by its GUID (globally unique identifier) and retrieve a temporary signed URL for object data in AWS or GCP that will provide direct access to that object. diff --git a/docs/introduction/default_expiration_times.md b/docs/additional_documentation/default_expiration_times.md similarity index 100% rename from docs/introduction/default_expiration_times.md rename to docs/additional_documentation/default_expiration_times.md diff --git a/docs/introduction/fence_create.md b/docs/additional_documentation/fence_create.md similarity index 100% rename from docs/introduction/fence_create.md rename to docs/additional_documentation/fence_create.md diff --git a/docs/introduction/setup.md b/docs/additional_documentation/setup.md similarity index 100% rename from docs/introduction/setup.md rename to docs/additional_documentation/setup.md diff --git a/docs/introduction/terminology.md b/docs/additional_documentation/terminology.md similarity index 100% rename from docs/introduction/terminology.md rename to docs/additional_documentation/terminology.md diff --git a/docs/introduction/token_management.md b/docs/additional_documentation/token_management.md similarity index 96% rename from docs/introduction/token_management.md rename to docs/additional_documentation/token_management.md index 01458716a..419a95d27 100644 --- a/docs/introduction/token_management.md +++ b/docs/additional_documentation/token_management.md @@ -1,7 +1,7 @@ ## Token management -Fence utilizes [OpenID Connect](./terminology.md#oidc) to generate tokens +Fence utilizes [OpenID Connect](terminology.mdidc) to generate tokens for clients. It can also provide tokens directly to a user. Clients and users may then use those tokens with other diff --git a/docs/introduction/oidc_and_oauth2.md b/docs/introduction/oidc_and_oauth2.md deleted file mode 100644 index b979a32e2..000000000 --- a/docs/introduction/oidc_and_oauth2.md +++ /dev/null @@ -1,74 +0,0 @@ -## OIDC & OAuth2 - -Fence acts as a central broker that supports multiple IdPs. -At the same time, it acts as an IdP itself. -In that sense, `fence` is both an `RP` and an `OP`. - -### Fence as RP - -Example: - -- Google IAM is the OpenID Provider (OP) -- Fence is the Relying Party (RP) -- Google Calendar API is the resource provider - -### Fence as OP - -- Fence is the OpenID Provider (OP) -- A third-party application is the Relying Party (RP) -- Gen3 microservices (e.g. [`sheepdog`](https://github.com/uc-cdis/sheepdog)) are resource providers - -### Example Flows - -Note that the `3rd Party App` acts as the `RP` in these examples. - -[//]: # (See /docs folder for README on how to regenerate these sequence diagrams) - -#### Flow: Client Registration - -![Client Registration](../images/seq_diagrams/client_registration.png) - -#### Flow: OpenID Connect - -In the following flow, Fence and the IdP together constitute an `OP`. -Fence, by itself, acts as an OAuth 2.0 Auth Server; the IdP enables the additional implementation of OIDC (by providing AuthN). From an OIDC viewpoint, therefore, Fence and the IdP can be abstracted into one `OP`. - -![OIDC Flow](../images/seq_diagrams/openid_connect_flow.png) - -If the third-party application doesn't need to use any Gen3 resources (and just -wants to authenticate the user), they can just get -needed information in the `ID token` after the handshake is finished . - -#### Flow: Using Tokens for Access - -If a third-party application wants to use Gen3 resources like -`fence`/`sheepdog`/`peregrine`, they call those services with an `Access Token` -passed in an `Authorization` header. - -In the following flow, `3rd Party App` is the `RP`; `Protected Endpoint` is an endpoint of a Gen3 Resource (the `microservice`), and both of these are part of a `resource server`; and `Fence` is the `OP`. Here, importantly, `Fence` may be interfacing with another IdP _or_ with another `Fence` instance in order to implement the OIDC layer. Either way, note that the `Fence` blob in this diagram actually abstracts Fence in concert with some IdP, which may or may not also be (a different instance of) Fence. - -![Using Access Token](../images/seq_diagrams/token_use_for_access.png) - -#### Flow: Refresh Token Use - -![Using Refresh Token](../images/seq_diagrams/refresh_token_use.png) - -#### Flow: Refresh Token Use (Token is Expired) - -![Using Expired Refresh Token](../images/seq_diagrams/refresh_token_use_expired.png) - -#### Flow: Multi-Tenant Fence - -The following diagram illustrates the case in which one fence instance -uses another fence instance as its identity provider. - -A use case for this is when we setup a fence instance that uses NIH login as the IdP. Here, we go through a detailed approval process in NIH. Therefore we would like to do it only once for a single lead Fence instance, and then allow other fence instances to simply redirect to use the lead Fence as an IdP for logging in via NIH. - -In the following flow, `Fence (Client Instance)` is an OP relative to `OAuth Client`, but an RP relative to `Fence (IDP)`. - -![Multi-Tenant Flow](../images/seq_diagrams/multi-tenant_flow.png) - -#### Notes - -See the [OIDC specification](http://openid.net/specs/openid-connect-core-1_0.html) for more details. -Additionally, see the [OAuth2 specification](https://tools.ietf.org/html/rfc6749). From 10eb8d3d536f28dc667aa51adbb10ec47cf0c513 Mon Sep 17 00:00:00 2001 From: Albert Snow Date: Tue, 23 Jul 2024 12:50:30 -0500 Subject: [PATCH 7/9] more minor fix, per alex's suggestion --- README.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 84bd4d22f..4f9ff2cae 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ and authorization for Gen3 services. 2. [Manage tokens](docs/additional_documentation/token_management.md). 3. Act as an [OIDC provider](README.md#oidc--oauth2) to support external applications to use Gen3 services. -4. [Issue short lived, cloud native credentials to access data in various cloud storage services](docs/additional_documentation/data_access.md#accessing-data) +4. [Issue short-lived, cloud native credentials to access data in various cloud storage services](docs/additional_documentation/data_access.md#accessing-data) ## Overview @@ -38,7 +38,7 @@ At the moment, supported IDPs include: - ORCID - RAS -### ## API Documentation +### API Documentation [OpenAPI documentation available here.](http://petstore.swagger.io/?url=https://raw.githubusercontent.com/uc-cdis/fence/master/openapis/swagger.yaml) @@ -110,7 +110,7 @@ In the following flow, `3rd Party App` is the `RP`; `Protected Endpoint` is an e The following diagram illustrates the case in which one fence instance uses another fence instance as its identity provider. -A use case for this is when we setup a fence instance that uses NIH login as the IdP. Here, we go through a detailed approval process in NIH. Therefore we would like to do it only once for a single lead Fence instance, and then allow other fence instances to simply redirect to use the lead Fence as an IdP for logging in via NIH. +A use case for this is when we set up a fence instance that uses NIH login as the IdP. Here, we go through a detailed approval process in NIH. Therefore, we would like to do it only once for a single lead Fence instance, and then allow other fence instances to simply redirect to use the lead Fence as an IdP for logging in via NIH. In the following flow, `Fence (Client Instance)` is an OP relative to `OAuth Client`, but an RP relative to `Fence (IDP)`. @@ -131,13 +131,12 @@ See [Fence and Google](docs/additional_documentation/google_architecture.md) for ## Setup -See detailed explination [here](docs/additional_documentation/setup.md) +See detailed explanation [here](docs/additional_documentation/setup.md) ## Additional documentation 1. [Terminologies](docs/additional_documentation/terminology.md) 2. [Accessing Data](docs/additional_documentation/data_access.md#accessing-data) -3. [Setup](docs/additional_documentation/setup.md) -4. [Token management](docs/additional_documentation/token_management.md) -5. [fence-create](docs/additional_documentation/fence_create.md) -6. [Default expiration times](docs/additional_documentation/default_expiration_times.md) +3. [Token management](docs/additional_documentation/token_management.md) +4. [fence-create](docs/additional_documentation/fence_create.md) +5. [Default expiration times](docs/additional_documentation/default_expiration_times.md) From 74c2e8244e88d73f352c4712dec2b415338f0458 Mon Sep 17 00:00:00 2001 From: Albert Snow Date: Tue, 23 Jul 2024 12:52:00 -0500 Subject: [PATCH 8/9] trailing spaces --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 4f9ff2cae..fc4ef7025 100644 --- a/README.md +++ b/README.md @@ -9,11 +9,11 @@ only trusted entities to enter. Fence is a core service of the Gen3 stack that has multiple capabilities: -1. Act as an [auth broker](docs/additional_documentation/terminology.md#auth-broker) to integrate with one +1. Act as an [auth broker](docs/additional_documentation/terminology.md#auth-broker) to integrate with one or more [IdPs](docs/additional_documentation/terminology.md#idp) and provide downstream authentication and authorization for Gen3 services. 2. [Manage tokens](docs/additional_documentation/token_management.md). -3. Act as an [OIDC provider](README.md#oidc--oauth2) to support external +3. Act as an [OIDC provider](README.md#oidc--oauth2) to support external applications to use Gen3 services. 4. [Issue short-lived, cloud native credentials to access data in various cloud storage services](docs/additional_documentation/data_access.md#accessing-data) From d459698aecdf0958fd4170cb65f223c6dfa8ce1a Mon Sep 17 00:00:00 2001 From: Albert Snow Date: Tue, 23 Jul 2024 12:59:51 -0500 Subject: [PATCH 9/9] dead links --- docs/additional_documentation/token_management.md | 2 +- docs/azure/azure_architecture.md | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/additional_documentation/token_management.md b/docs/additional_documentation/token_management.md index 419a95d27..c8c39eba5 100644 --- a/docs/additional_documentation/token_management.md +++ b/docs/additional_documentation/token_management.md @@ -1,7 +1,7 @@ ## Token management -Fence utilizes [OpenID Connect](terminology.mdidc) to generate tokens +Fence utilizes [OpenID Connect](terminology.md#oidc) to generate tokens for clients. It can also provide tokens directly to a user. Clients and users may then use those tokens with other diff --git a/docs/azure/azure_architecture.md b/docs/azure/azure_architecture.md index a7bf1d170..acf15ff55 100755 --- a/docs/azure/azure_architecture.md +++ b/docs/azure/azure_architecture.md @@ -26,7 +26,7 @@ Also note that there's alternatives that could be considered for [future develop You can provision an Azure Storage Account with [Blob Storage as a one-time setup](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal). You can further make the blobs [public read only](https://docs.microsoft.com/en-us/azure/storage/blobs/anonymous-read-access-configure?tabs=portal) for dev / test purposes, but it would be advisable to avoid this setup in a production scenario and consider using non-public ones instead. -With the Azure Blob Storage Account setup, you can further upload files into your Blob Storage using [Azure Storage Explorer](https://azure.microsoft.com/en-us/features/storage-explorer/) for manual testing, or you can automate loading data files into Azure Blob Storage (e.g. using [Azure Blob Storage Python SDK](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-blob)). +With the Azure Blob Storage Account setup, you can further upload files into your Blob Storage using [Azure Storage Explorer](https://azure.microsoft.com/en-us/products/storage/storage-explorer/) for manual testing, or you can automate loading data files into Azure Blob Storage (e.g. using [Azure Blob Storage Python SDK](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/azure-storage-blob)). Assuming that you have preexisting files in an Azure Blob Storage Account, you can work through the following steps to index the files: @@ -46,10 +46,10 @@ You can use the Azure Blob Storage client to connect to Azure Blob Storage, and You can update the [Fence config.yaml](../../fence/config-default.yaml) to include the following values: -Name | Value | Description ------- | ------|---------- -`AZ_BLOB_CREDENTIALS` | DefaultEndpointsProtocol=https;AccountName=somestorageaccount;AccountKey=storageaccountkey;BlobEndpoint=`https://somestorageaccount.blob.core.windows.net/`; | This is the [Azure Blob Storage Connection String](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&tabs=azure-portal#view-account-access-keys). You can also set this to `'*'` if you are indexing URLs for [public read access Azure Blob Storage containers](https://docs.microsoft.com/en-us/azure/storage/blobs/anonymous-read-access-configure?tabs=portal). Note that if you're using the URL for a public read access Azure Blob Storage container, then operations such as `delete` and `upload` will not work. -`AZ_BLOB_CONTAINER_URL` | `https://storageaccountname.blob.core.windows.net/storage-container` | This is the destination container for uploading with a given SAS token. You can set this value to designate a pre-existing storage container to upload indexed files, for example the new files could sit in `https://storageaccountname.blob.core.windows.net/storage-container/someguid/some/blob/file.txt`. If the storage account doesn't align with the indexed URL (e.g. you're using a public url or the storage account doesn't match), the upload will not work. If `AZ_BLOB_CREDENTIALS` is `'*'` then uploads from an indexed file using a public URL will not work. This value should be associated with the same Azure Blob Storage account used with the [Azure Blob Storage Connection String](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&tabs=azure-portal#view-account-access-keys) for `AZ_BLOB_CREDENTIALS`. +| Name | Value | Description | +|-------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `AZ_BLOB_CREDENTIALS` | DefaultEndpointsProtocol=https;AccountName=somestorageaccount;AccountKey=storageaccountkey;BlobEndpoint=`https://somestorageaccount.blob.core.windows.net/`; | This is the [Azure Blob Storage Connection String](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&tabs=azure-portal#view-account-access-keys). You can also set this to `'*'` if you are indexing URLs for [public read access Azure Blob Storage containers](https://docs.microsoft.com/en-us/azure/storage/blobs/anonymous-read-access-configure?tabs=portal). Note that if you're using the URL for a public read access Azure Blob Storage container, then operations such as `delete` and `upload` will not work. | +| `AZ_BLOB_CONTAINER_URL` | `https://storageaccountname.blob.core.windows.net/storage-container` | This is the destination container for uploading with a given SAS token. You can set this value to designate a pre-existing storage container to upload indexed files, for example the new files could sit in `https://storageaccountname.blob.core.windows.net/storage-container/someguid/some/blob/file.txt`. If the storage account doesn't align with the indexed URL (e.g. you're using a public url or the storage account doesn't match), the upload will not work. If `AZ_BLOB_CREDENTIALS` is `'*'` then uploads from an indexed file using a public URL will not work. This value should be associated with the same Azure Blob Storage account used with the [Azure Blob Storage Connection String](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&tabs=azure-portal#view-account-access-keys) for `AZ_BLOB_CREDENTIALS`. | Using pre-signed urls for download is implemented; it's currently using a [SAS Token](https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview).