Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doc Updates #159

Merged
merged 47 commits into from
Oct 31, 2024
Merged
Show file tree
Hide file tree
Changes from 44 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
5bda45f
Move files from local
yanmastin-astro Oct 18, 2024
deaf6fc
Commit generated docs
yanmastin-astro Oct 18, 2024
4e0559c
Updates from review
yanmastin-astro Oct 21, 2024
6c5dddc
Update generated docs
yanmastin-astro Oct 21, 2024
179deeb
Update templates/guides/import-script.md
yanmastin-astro Oct 22, 2024
4facb11
Updates from review
yanmastin-astro Oct 22, 2024
03cecc1
Formatting fixes
yanmastin-astro Oct 22, 2024
4c88e5f
Update generated docs
yanmastin-astro Oct 22, 2024
e18ca2b
Update templates/guides/get-started.md
yanmastin-astro Oct 23, 2024
f55c12c
Update templates/guides/get-started.md
yanmastin-astro Oct 23, 2024
10873a1
Update templates/guides/get-started.md
yanmastin-astro Oct 23, 2024
1f4eac3
Update templates/guides/import-script.md
yanmastin-astro Oct 23, 2024
a7c9ffb
Update templates/guides/import-script.md
yanmastin-astro Oct 23, 2024
735720f
Update templates/guides/import-script.md
yanmastin-astro Oct 23, 2024
ae35d61
Update templates/guides/get-started.md
yanmastin-astro Oct 23, 2024
0983feb
Update templates/guides/import-script.md
yanmastin-astro Oct 23, 2024
0206161
Update templates/guides/import-script.md
yanmastin-astro Oct 23, 2024
cd56eb5
Update from review
yanmastin-astro Oct 23, 2024
e7053f0
Add prereqs
yanmastin-astro Oct 23, 2024
5a9f76b
Add note
yanmastin-astro Oct 23, 2024
b7ca225
Update generated docs
yanmastin-astro Oct 23, 2024
50291fc
Misc update
yanmastin-astro Oct 25, 2024
a2b4a7e
Update templates/guides/import-script.md
yanmastin-astro Oct 25, 2024
c1c6aff
Update templates/guides/get-started.md
yanmastin-astro Oct 25, 2024
9431cf7
Update templates/guides/get-started.md
yanmastin-astro Oct 25, 2024
87ee5a8
Update templates/guides/get-started.md
yanmastin-astro Oct 25, 2024
ce179de
Update templates/guides/get-started.md
yanmastin-astro Oct 25, 2024
d549319
Update templates/guides/import-script.md
yanmastin-astro Oct 25, 2024
9b51a95
Update templates/guides/import-script.md
yanmastin-astro Oct 25, 2024
2066346
Update templates/guides/import-script.md
yanmastin-astro Oct 25, 2024
6e14481
Update templates/guides/import-script.md
yanmastin-astro Oct 25, 2024
6a6bcf2
Add clarity to note
yanmastin-astro Oct 25, 2024
a4e5cc4
Remove description
yanmastin-astro Oct 28, 2024
7e7407d
Update generated docs
yanmastin-astro Oct 28, 2024
ec5d92b
Update templates/guides/import-script.md
yanmastin-astro Oct 29, 2024
84d78f1
Update templates/guides/import-script.md
yanmastin-astro Oct 30, 2024
c254fbe
Update templates/guides/import-script.md
yanmastin-astro Oct 30, 2024
cc81d65
Update templates/guides/get-started.md
yanmastin-astro Oct 30, 2024
1d69fb1
Update templates/guides/import-script.md
yanmastin-astro Oct 30, 2024
2288384
Update templates/guides/get-started.md
yanmastin-astro Oct 30, 2024
237ce3f
Update templates/guides/get-started.md
yanmastin-astro Oct 30, 2024
13f6d20
Make active voice
yanmastin-astro Oct 30, 2024
a07942c
Update generated docs
yanmastin-astro Oct 30, 2024
3dbb93f
Update generated docs
yanmastin-astro Oct 30, 2024
ccc558d
Misc update
yanmastin-astro Oct 31, 2024
8e6e32c
Update templates/guides/import-script.md
yanmastin-astro Oct 31, 2024
a07c495
Update generated docs
yanmastin-astro Oct 31, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
223 changes: 223 additions & 0 deletions docs/guides/get-started.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,223 @@
---
page_title: "Get started with Astro Terraform Provider"
---

# Get started with Astro Terraform Provider
This guide shows you how to automate the onboarding of a new team onto Astro by creating and managing a Workspace and Deployment. By the end of this tutorial, you will have a fully automated setup that is reproducible and easily scalable to more teams.

## Prerequisites
- An Astro Organization
- [Terraform](https://developer.hashicorp.com/terraform/install)


## Step 1: Create Your Terraform Working Directory
1. Create a folder, `my-data-platform` for your Terraform project.
2. Save the following code in a file named `terraform.tf`:
```
terraform {
required_providers {
astro = {
source = "astronomer/astro"
version = "1.0.0"
}
}
}

provider "astro" {
organization_id = <your-organization-id>
}
```
3. Edit the `terraform.tf` file with your Organization's ID for `<your-organization-id>`. The working directory contains all your Terraform code, and all Terraform commands run from this directory.

## Step 2: Initialize the Terraform Working Directory
1. Run `terraform init`. You will see Terraform downloading and installing the Astro Terraform provider locally:
```
$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding astronomer/astro versions matching "1.0.0"...
- Installing astronomer/astro v1.0.0...
- Installed astronomer/astro v1.0.0 (signed by a HashiCorp partner, key ID F5206453FDEA33CF)

...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
```
2. Terraform stores the versions and hashes of providers in a generated file `.terraform.lock.hcl`.

-> Astronomer recommends you store `.terraform.lock.hcl` in your version control system so that you can track potential changes to your external dependencies.

## Step 3: Authenticate with Astro
1. [Create an API token](https://www.astronomer.io/docs/astro/automation-authentication#step-1-create-an-api-token) in Astro. Since you are creating a Workspace, you need an [Organization API token](https://www.astronomer.io/docs/astro/organization-api-tokens) with [Organization Owner permissions](https://www.astronomer.io/docs/astro/user-permissions#organization-roles).
2. Configure the API token as an environment variable `ASTRO_API_TOKEN` to run Terraform commands:
`export ASTRO_API_TOKEN=<your-api-token>`

Alternatively, you can set their API token value in the provider block:
```
provider "astro" {
organization_id = "&lt;your-organization-id&gt;"
token = "&lt;your-api-token&gt;"
}
```

## Step 4: Define Resources in Terraform
In a file `main.tf`, define two resources, an `astro_workspace` and an `astro_deployment`. These resources represent an Astro Workspace and Astro Deployment, defined in Terraform code:
```
# Create a new workspace
resource "astro_workspace" "my_first_tf_workspace" {
name = "My first TF workspace"
description = "My first Terraform-created workspace"
cicd_enforced_default = false
}

# Create a new Deployment
resource "astro_deployment" "my_first_tf_deployment" {
name = "My first TF deployment"
description = "My first Terraform-created deployment"
type = "STANDARD"
cloud_provider = "AWS"
region = "us-east-1"
contact_emails = []
default_task_pod_cpu = "0.25"
default_task_pod_memory = "0.5Gi"
executor = "CELERY"
is_cicd_enforced = true
is_dag_deploy_enabled = true
is_development_mode = false
is_high_availability = false
resource_quota_cpu = "10"
resource_quota_memory = "20Gi"
scheduler_size = "SMALL"
workspace_id = astro_workspace.my_first_tf_workspace.id
environment_variables = []
worker_queues = [{
name = "default"
is_default = true
astro_machine = "A5"
max_worker_count = 10
min_worker_count = 0
worker_concurrency = 1
}]
}
```
-> One of the key characteristics (and benefits) of using Terraform is that it's *declarative*. For example, `workspace_id = astro_workspace.my_first_tf_workspace.id` tells Terraform to configure the Workspace ID in the Deployment. This means the Workspace must be created first, producing an ID which is a generated value and unknown at the time you initially define the resources in Terraform. You don't have to instruct Terraform to create resources in a certain order, you only have to instruct what to create. You can define the resources in the previous code example in any order. Terraform takes the relationships between resources into account when deciding the order of creating resources.

## Step 5: (Optional) Define Outputs
In a file called `outputs.tf`, define values you want to log after creating the infrastructure. The following code configures Workspace and Deployment IDs as the output:
```
output "terraform_workspace" {
description = "ID of the TF created workspace"
value = astro_workspace.my_first_tf_workspace.id
}

output "terraform_deployment" {
description = "ID of the TF created deployment"
value = astro_deployment.my_first_tf_deployment.id
}
```

## Step 6: Preview the Terraform changes
You now have up to three files: `terraform.tf`, `main.tf`, and the optional `outputs.tf`.
1. Run [`terraform plan`](https://developer.hashicorp.com/terraform/cli/commands/plan) to let Terraform create an execution plan, so you can preview the infrastructure changes that Terraform will make. As Terraform makes a plan, it shares a summary of the plan as an output similar to the following text:
```
$ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

# astro_deployment.my_first_tf_deployment will be created
+ resource "astro_deployment" "my_first_tf_deployment" {
...
}

# astro_workspace.my_first_tf_workspace will be created
+ resource "astro_workspace" "my_first_tf_workspace" {
...
}

Plan: 2 to add, 0 to change, 0 to destroy.

Changes to Outputs:
+ terraform_deployment = (known after apply)
+ terraform_workspace = (known after apply)
```
2. Verify that the generated plan contains the text `Plan: 2 to add, 0 to change, 0 to destroy.` This confirms that `main.tf` creates two new resources. In this case, a Workspace and a Deployment.

## Step 7: Apply the Terraform Plan
Run `terraform apply` and select `yes` to execute the plan. This creates the Astro resources and prints their ids, as you defined in `outputs.tf`:
```
$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

...

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

astro_workspace.my_first_tf_workspace: Creating...
astro_workspace.my_first_tf_workspace: Creation complete after 0s [id=&lt;workspace-id&gt]
astro_deployment.my_first_tf_deployment: Creating...
astro_deployment.my_first_tf_deployment: Creation complete after 1s [id=&lt;deployment-id&gt]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

terraform_deployment = "&lt;deployment-id&gt"
terraform_workspace = "&lt;workspace-id&gt"
```
The resources were created and are now visible in Astro.

## Step 8: Clean Up Terraform-Created Resources
Run `terraform destroy` and select `yes`:
```
$ terraform destroy
astro_workspace.my_first_tf_workspace: Refreshing state...
[id=&lt;workspace-id&gt]
astro_deployment.my_first_tf_deployment: Refreshing state...
[id=&lt;deployment-id&gt]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
- destroy

Terraform will perform the following actions:

...

Plan: 0 to add, 0 to change, 2 to destroy.

Changes to Outputs:
- terraform_deployment = "&lt;deployment-id&gt" -> null
- terraform_workspace = "&lt;workspace-id&gt" -> null

Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.

Enter a value: yes

astro_deployment.my_first_tf_deployment: Destroying...
[id=&lt;deployment-id&gt]
astro_deployment.my_first_tf_deployment: Destruction complete after 1s
astro_workspace.my_first_tf_workspace: Destroying...
[id=&lt;workspace-id&gt]
astro_workspace.my_first_tf_workspace: Destruction complete after 0s

Destroy complete! Resources: 2 destroyed.
```
The output shows two destroyed resources, which are the Workspace and Deployment that you first created.
132 changes: 132 additions & 0 deletions docs/guides/import-script.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
---
page_title: "Use Terraform Import Script to migrate existing resources"
---

# Use Import Script to migrate existing resources
The Astro Terraform Import Script helps you import existing Astro resources into your Terraform configuration.

This guide shows you how to migrate an existing Workspace, API token, and Team into Terraform using the Terraform Import Script.

## Import Script options
- `-resources`: Comma-separated list of resources to import. Accepted values are
`workspace`, `deployment`, `cluster`, `api_token`, `team`, `team_roles`,and `user_roles`. If not provided, all resources are imported.

-> Ensure you have the necessary permissions in your Astro Organization to access the resources you're attempting to import. See [Astro User Permissions Reference](https://www.astronomer.io/docs/astro/user-permissions) for more information.

- `-token`: API token to authenticate with the Astro platform. If not provided, the script will attempt to use the `ASTRO_API_TOKEN` environment variable.
- `-organizationId`: Organization ID to import resources from.
- `-runTerraformInit`: Run `terraform init` after generating the import configuration. Used for initializing the Terraform state in our GitHub Actions.
- `-help`: Display help information.


## Prerequisites
- An [Astro](https://www.astronomer.io/product/) Organization with a Workspace, Team, and API token
- An initialized Terraform working directory

## Step 1: Download the Import Script
1. Download the `terraform-provider-astro-import-script` executable file from the [Astro Terraform Provider releases](https://github.com/astronomer/terraform-provider-astro/releases) based on your OS and architecture. For this guide, the script will be `terraform-provider-astro-import-script_v0.1.3_darwin_arm64`.

## Step 2: Run the Import Script

-> Make sure you run `terraform init` before using the Import Script, or use the `-runTerraformInit` option when running the Import Script.

1. Authenticate with Astro by creating an [API token](https://www.astronomer.io/docs/astro/organization-api-tokens#create-an-organization-api-token) with the **Organization owner** role and configure it as an `ASTRO_API_TOKEN` environment variable:
```
export ASTRO_API_TOKEN=&lt;your-api-token&gt;
```

2. If you are using a Unix-based systems, add execute permission to the script file:
```
chmod +x terraform-provider-astro-import-script_&lt;version-number&gt;_&lt;os&gt;_&lt;arc&gt;
```
3. Run the Import Script. Insert the script's version, your computer's operating system, and your computer's architecture for `<version-number>`, `<os>`, and `<arc>`.

- On Unix-based systems:
```
./terraform-provider-astro-import-script_&lt;version-number&gt;_&lt;os&gt;_&lt;arc&gt; [options]
```
- On Windows:

```
.\terraform-provider-astro-import-script_&lt;version-number&gt;_&lt;os&gt;_&lt;arc&gt;.exe [options]
```

To import your existing Workspace, API token and Team, specify those resources with the `-resources` option. The other option you need to specify is `-organizationId`:
```
./terraform-provider-astro-import-script_v0.1.3_darwin_arm64 -organizationId &lt;your-organization-id&gt; -resources api_token,team,workspace
```

You should see the following output:
```
Terraform Import Script Starting
Resources to import: [api_token team workspace]
Using organization ID: &lt;your-organization-id&gt
Terraform version 1.9.7 is installed and meets the minimum required version.
Importing teams for organization &lt;your-organization-id&gt
Importing API tokens for organization &lt;your-organization-id&gt
Importing workspaces for organization &lt;your-organization-id&gt
Importing Workspaces: [&lt;workspace-id&gt]
Successfully handled resource workspace
Importing API Tokens: [&lt;api_token-id&gt]
Successfully handled resource api_token
Importing Teams: [&lt;team-id&gt]
Successfully handled resource team
Successfully wrote import configuration to import.tf
Successfully deleted generated.tf
terraform.tfstate does not exist, no need to delete
astro_team.team_&lt;team-id&gt: Preparing import... [id=&lt;team-id&gt]
astro_api_token.api_token_&lt;api_token-id&gt: Preparing import... [id=&lt;api_token-id&gt]
astro_workspace.workspace_&lt;workspace-id&gt: Preparing import... [id=&lt;workspace-id&gt]
astro_workspace.workspace_&lt;workspace-id&gt: Refreshing state... [id=&lt;workspace-id&gt]
astro_api_token.api_token_&lt;api_token-id&gt: Refreshing state... [id=&lt;api_token-id&gt]
astro_team.team_&lt;team-id&gt: Refreshing state... [id=&lt;team-id&gt]

Terraform will perform the following actions:

...

Plan: 3 to import, 0 to add, 0 to change, 0 to destroy.

Terraform has generated configuration and written it to generated.tf. Please review the configuration and edit it as
necessary before adding it to version control.

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you
run "terraform apply" now.
Import process completed. Summary:
Resource workspace processed successfully
Resource api_token processed successfully
Resource team processed successfully
```
-> If you import Deployments, they don't count towards the `Plan: 3 to import, 0 to add, 0 to change, 0 to destroy` line of the output, even when the Deployments are successfully imported. This is a known issue and is in the process of being fixed.

## Step 3: Review output
The script generates two main files:
- `import.tf`: Contains the Terraform import blocks for the specified resources.
- `generated.tf`: Contains the Terraform resource configurations for the imported resources.
The generated Terraform configurations might require some manual adjustment to match your specific requirements or to resolve any conflicts.

## Step 4: Extract and organize resources
The `generated.tf` file created by the Import Script contains all of the specified resources in one file. Astronomer recommends that you extract and modularize the resources so they are easily maintained and reusable. The following example shows a well structured Terraform project for managing Astro infrastructure:
```
terraform-astro-project/
├── environments/
│ ├── dev/
│ │ ├── main.tf # Root module for development
│ │ ├── variables.tf # Dev-specific variables
│ │ ├── outputs.tf # Dev-specific outputs
│ │ └── dev.tfvars # Variable values for development
│ ├── prod/
│ │ ├── main.tf # Root module for production
│ │ ├── variables.tf # Prod-specific variables
│ │ ├── outputs.tf # Prod-specific outputs
│ │ └── prod.tfvars # Variable values for production
├── modules/
│ ├── astro/
│ │ ├── main.tf # Entry point for the module
│ │ ├── workspace.tf # Defines Astro workspaces
│ │ ├── deployment.tf # Defines Astro Deployments
│ │ ├── users.tf # Defines user roles and access
│ │ ├── variables.tf # Variables used in the astro module
│ │ └── outputs.tf # Outputs from the astro module
└── cloud_provider.tf
```
22 changes: 12 additions & 10 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,19 @@
---
# generated by https://github.com/hashicorp/terraform-plugin-docs
page_title: "astro Provider"
subcategory: ""
description: |-

page_title: "Astro Provider"
---

# astro Provider
# Astro Provider
[Astro](https://www.astronomer.io/product/) is a unified data platform built on Apache Airflow® that ensures data is delivered on time, securely, and accurately.
This is the official [Astro Terraform Provider](https://github.com/astronomer/terraform-provider-astro), built to allow you to automate, scale, and manage your Astro infrastructure.
Reduce manual effort, minimize errors, and create a more streamlined approach to managing data orchestration.

## Authentication
Create an [API token](https://www.astronomer.io/docs/astro/automation-authentication#step-1-create-an-api-token) with the minimum required [permissions](https://www.astronomer.io/docs/astro/user-permissions) to securely use Terraform, ensuring it only has access to the resources necessary.
You can create a [Deployment](https://www.astronomer.io/docs/astro/deployment-api-tokens), [Workspace](https://www.astronomer.io/docs/astro/workspace-api-tokens), or [Organizaton](https://www.astronomer.io/docs/astro/organization-api-tokens) API token.
An Organizaton token is the most flexible option to authenticate for high level changes accross multiple different resources.
Ensure the API token is configured as an environment variable, `ASTRO_API_TOKEN` when running Terraform commands.


## Example Usage

## Example usage
```terraform
provider "astro" {
organization_id = "cljzz64cc001n01mln1p12345"
Expand All @@ -28,4 +30,4 @@ provider "astro" {
### Optional

- `host` (String) API host to use for the provider. Default is `https://api.astronomer.io`
- `token` (String, Sensitive) Astro API Token. Can be set with an `ASTRO_API_TOKEN` env var.
- `token` (String, Sensitive) Astro API Token. Can be set with an `ASTRO_API_TOKEN` env var.
Loading