Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keep updating v5 #106

Open
wants to merge 14 commits into
base: master
Choose a base branch
from
Open

Keep updating v5 #106

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
68a044b
Keep updating v5 for catalogs
Michaelvll Dec 13, 2024
5d21c9d
fix latest commit
Michaelvll Dec 13, 2024
b14c85e
Add venv
Michaelvll Dec 13, 2024
11c574b
test with ref
Michaelvll Dec 13, 2024
46bd99a
fix
Michaelvll Dec 13, 2024
e09d9e0
authorize with github action for GCP
Michaelvll Dec 13, 2024
31dd9f5
Add commit hash
Michaelvll Dec 13, 2024
fdc72df
fix commit hash fetcher
Michaelvll Dec 13, 2024
9f6120d
add history for sky
Michaelvll Dec 13, 2024
ed3bf1d
cd sky
Michaelvll Dec 13, 2024
0d29879
move sky
Michaelvll Dec 13, 2024
97e6ce4
fix
Michaelvll Dec 13, 2024
989a462
fix versions
Michaelvll Dec 13, 2024
84eb1a2
Address comment
Michaelvll Dec 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 31 additions & 17 deletions .github/workflows/update-aws-catalog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ jobs:
- name: Clone SkyPilot repo
uses: actions/checkout@v4
with:
fetch-depth: 0
repository: skypilot-org/skypilot
path: sky
- name: Clone Catalog repo
Expand All @@ -24,36 +25,49 @@ jobs:
fetch-depth: 0
path: catalogs
token: ${{ secrets.GH_ACTION_PAT }}
- name: Set up Python 3.10
uses: actions/setup-python@v5
- name: Install the latest version of uv
uses: astral-sh/setup-uv@v4
with:
python-version: '3.10'
cache: 'pip' # caching pip dependencies
- name: Install dependencies
run: |
python -m pip install --upgrade pip
cd sky
pip install ".[aws]"

version: "latest"
python-version: "3.10"
- name: Run fetch_aws
id: fetch_catalogs
run: |
version=$(python -c 'import sky; print(sky.clouds.service_catalog.constants.CATALOG_SCHEMA_VERSION)')
mkdir -p catalogs/catalogs/$version
cd catalogs/catalogs/$version
# Uses --check-all-regions-enabled-for-account to ensure that the catalog fetched has all regions, otherwise fail the job.
python -m sky.clouds.service_catalog.data_fetchers.fetch_aws --no-az-mappings --check-all-regions-enabled-for-account
uv venv --seed ~/catalogs-venv
source ~/catalogs-venv/bin/activate
versions=$(cat catalogs/.metadata.yml | yq -r '.version_commit | keys[]')
# Loop over all versions and commit hashes
for version in $versions; do
# Find commit hash for the version
commit_hash=$(cat catalogs/.metadata.yml | yq -r ".version_commit.$version")
cd sky
if [ "$commit_hash" == "latest" ]; then
# Find the latest commit of the remote SkyPilot repo
commit_hash=$(git rev-parse origin/master)
fi
git checkout $commit_hash
uv pip install ".[aws]"
cd -
echo "Fetching AWS catalog for version $version, SkyPilot commit hash $commit_hash"
mkdir -p catalogs/catalogs/$version
cd catalogs/catalogs/$version
# Uses --check-all-regions-enabled-for-account to ensure that the catalog fetched has all regions, otherwise fail the job.
python -m sky.clouds.service_catalog.data_fetchers.fetch_aws --no-az-mappings --check-all-regions-enabled-for-account
cd -
done
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

- name: Commit catalog
run: |
version=$(python -c 'import sky; print(sky.clouds.service_catalog.constants.CATALOG_SCHEMA_VERSION)')
versions=$(cat catalogs/.metadata.yml | yq -r '.version_commit | keys[]')
cd catalogs
git config --global user.name 'github-actions[bot]'
git config --global user.email 'github-actions[bot]@users.noreply.github.com'
git add .
git commit -m"[Bot] Update AWS catalog $version (scheduled at $(date))" || { echo "No changes to commit" && exit 0; }
git commit -m"[Bot] Update AWS catalog $versions (scheduled at $(date))" || { echo "No changes to commit" && exit 0; }
git fetch origin
git rebase origin/master
git push

42 changes: 28 additions & 14 deletions .github/workflows/update-fluidstack-catalog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ jobs:
- name: Clone SkyPilot repo
uses: actions/checkout@v4
with:
fetch-depth: 0
repository: skypilot-org/skypilot
path: sky
- name: Clone Catalog repo
Expand All @@ -23,35 +24,48 @@ jobs:
fetch-depth: 0
path: catalogs
token: ${{ secrets.GH_ACTION_PAT }}
- name: Set up Python 3.10
uses: actions/setup-python@v5
- name: Install the latest version of uv
uses: astral-sh/setup-uv@v4
with:
version: "latest"
python-version: "3.10"
cache: "pip" # caching pip dependencies
- name: Install dependencies
run: |
python -m pip install --upgrade pip
uv venv --seed ~/catalogs-venv
source ~/catalogs-venv/bin/activate
cd sky
pip install .

uv pip install ".[fluidstack]"
- name: Run fetch_fluidstack
id: fetch_catalogs
run: |
version=$(python -c 'import sky; print(sky.clouds.service_catalog.constants.CATALOG_SCHEMA_VERSION)')
mkdir -p catalogs/catalogs/$version
cd catalogs/catalogs/$version
mkdir -p ~/.fluidstack
echo "$FLUIDSTACK_API_KEY" > ~/.fluidstack/api_key
python -m sky.clouds.service_catalog.data_fetchers.fetch_fluidstack
source ~/catalogs-venv/bin/activate
versions=$(cat catalogs/.metadata.yml | yq -r '.version_commit | keys[]')
for version in $versions; do
commit_hash=$(cat catalogs/.metadata.yml | yq -r ".version_commit.$version")
cd sky
if [ "$commit_hash" == "latest" ]; then
commit_hash=$(git rev-parse origin/master)
fi
git checkout $commit_hash
uv pip install ".[fluidstack]"
cd -
mkdir -p catalogs/catalogs/$version
cd catalogs/catalogs/$version
mkdir -p ~/.fluidstack
echo "$FLUIDSTACK_API_KEY" > ~/.fluidstack/api_key
python -m sky.clouds.service_catalog.data_fetchers.fetch_fluidstack
cd -
done
env:
FLUIDSTACK_API_KEY: ${{ secrets.FLUIDSTACK_API_KEY }}
- name: Commit catalog
run: |
version=$(python -c 'import sky; print(sky.clouds.service_catalog.constants.CATALOG_SCHEMA_VERSION)')
versions=$(cat catalogs/.metadata.yml | yq -r '.version_commit | keys[]')
cd catalogs
git config --global user.name 'github-actions[bot]'
git config --global user.email 'github-actions[bot]@users.noreply.github.com'
git add .
git commit -m"[Bot] Update FluidStack catalog $version (scheduled at $(date))" || { echo "No changes to commit" && exit 0; }
git commit -m"[Bot] Update FluidStack catalog $versions (scheduled at $(date))" || { echo "No changes to commit" && exit 0; }
git fetch origin
git rebase origin/master
git push
65 changes: 35 additions & 30 deletions .github/workflows/update-gcp-catalog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ jobs:
- name: Clone SkyPilot repo
uses: actions/checkout@v4
with:
fetch-depth: 0
repository: skypilot-org/skypilot
path: sky
- name: Clone Catalog repo
Expand All @@ -24,53 +25,57 @@ jobs:
fetch-depth: 0
path: catalogs
token: ${{ secrets.GH_ACTION_PAT }}
- name: Set up Python 3.10
uses: actions/setup-python@v5
- name: Install the latest version of uv
uses: astral-sh/setup-uv@v4
with:
python-version: '3.10'
cache: 'pip' # caching pip dependencies
version: "latest"
python-version: "3.10"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
uv venv --seed ~/catalogs-venv
source ~/catalogs-venv/bin/activate
cd sky
pip install ".[gcp]" &
uv pip install ".[gcp]"
cd ~
curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-436.0.0-linux-x86_64.tar.gz
tar -xf google-cloud-cli-436.0.0-linux-x86_64.tar.gz
./google-cloud-sdk/install.sh -q
wait

- id: 'auth'
uses: 'google-github-actions/auth@v2'
with:
credentials_json: '${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}'
- name: Run fetch_gcp
id: fetch_catalogs
run: |
echo "> Sourcing gcloud path ..."
source ~/google-cloud-sdk/path.bash.inc
echo "> Writing service account key to file ..."
echo "$GCP_SERVICE_ACCOUNT_KEY" > $HOME/service_account_key.json
echo "> Exporting service account key ..."
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/service_account_key.json
echo "> Activating service account ..."
gcloud auth activate-service-account --key-file $GOOGLE_APPLICATION_CREDENTIALS > /dev/null
echo "> Configuring project ..."
gcloud config set project $GCP_PROJECT_ID > /dev/null
echo "> Fetching catalog schema version ..."
version=$(python -c 'import sky; print(sky.clouds.service_catalog.constants.CATALOG_SCHEMA_VERSION)')
echo "> Creating catalog directory ..."
mkdir -p catalogs/catalogs/$version
cd catalogs/catalogs/$version
echo "> Fetching GCP catalog ..."
python -u -m sky.clouds.service_catalog.data_fetchers.fetch_gcp --all-regions --single-threaded
source ~/catalogs-venv/bin/activate
versions=$(cat catalogs/.metadata.yml | yq -r '.version_commit | keys[]')
for version in $versions; do
commit_hash=$(cat catalogs/.metadata.yml | yq -r ".version_commit.$version")
cd sky
echo "> Fetching catalog schema version $version with SkyPilot commit hash $commit_hash ..."
if [ "$commit_hash" == "latest" ]; then
commit_hash=$(git rev-parse origin/master)
fi
echo "> Checking out SkyPilot to commit hash $commit_hash ..."
git checkout $commit_hash
uv pip install ".[gcp]"
cd -
echo "> Creating catalog directory ..."
mkdir -p catalogs/catalogs/$version
cd catalogs/catalogs/$version
echo "> Fetching GCP catalog ..."
python -u -m sky.clouds.service_catalog.data_fetchers.fetch_gcp --all-regions --single-threaded
cd -
done
env:
GCP_SERVICE_ACCOUNT_KEY: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}

- name: Commit catalog
run: |
version=$(python -c 'import sky; print(sky.clouds.service_catalog.constants.CATALOG_SCHEMA_VERSION)')
versions=$(cat catalogs/.metadata.yml | yq -r '.version_commit | keys[]')
cd catalogs
git config --global user.name 'github-actions[bot]'
git config --global user.email 'github-actions[bot]@users.noreply.github.com'
git add .
git commit -m"[Bot] Update GCP catalog $version (scheduled at $(date))" || { echo "No changes to commit" && exit 0; }
git commit -m"[Bot] Update GCP catalog $versions (scheduled at $(date))" || { echo "No changes to commit" && exit 0; }
git fetch origin
git rebase origin/master
git push
39 changes: 27 additions & 12 deletions .github/workflows/update-lambda-catalog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ jobs:
- name: Clone SkyPilot repo
uses: actions/checkout@v4
with:
fetch-depth: 0
repository: skypilot-org/skypilot
path: sky
- name: Clone Catalog repo
Expand All @@ -24,34 +25,48 @@ jobs:
fetch-depth: 0
path: catalogs
token: ${{ secrets.GH_ACTION_PAT }}
- name: Set up Python 3.10
uses: actions/setup-python@v5
- name: Install the latest version of uv
uses: astral-sh/setup-uv@v4
with:
python-version: '3.10'
cache: 'pip' # caching pip dependencies
version: "latest"
python-version: "3.10"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
uv venv --seed ~/catalogs-venv
source ~/catalogs-venv/bin/activate
cd sky
pip install ".[lambda]"
uv pip install ".[lambda]"

- name: Run fetch_lambda
id: fetch_catalogs
run: |
version=$(python -c 'import sky; print(sky.clouds.service_catalog.constants.CATALOG_SCHEMA_VERSION)')
mkdir -p catalogs/catalogs/$version
cd catalogs/catalogs/$version
python -m sky.clouds.service_catalog.data_fetchers.fetch_lambda_cloud --api-key ${LAMBDA_API_KEY}
source ~/catalogs-venv/bin/activate
versions=$(cat catalogs/.metadata.yml | yq -r '.version_commit | keys[]')
for version in $versions; do
commit_hash=$(cat catalogs/.metadata.yml | yq -r ".version_commit.$version")
cd sky
if [ "$commit_hash" == "latest" ]; then
commit_hash=$(git rev-parse origin/master)
fi
git checkout $commit_hash
uv pip install ".[lambda]"
cd -
mkdir -p catalogs/catalogs/$version
cd catalogs/catalogs/$version
python -m sky.clouds.service_catalog.data_fetchers.fetch_lambda_cloud --api-key ${LAMBDA_API_KEY}
cd -
done
env:
LAMBDA_API_KEY: ${{ secrets.LAMBDA_API_KEY }}

- name: Commit catalog
run: |
version=$(python -c 'import sky; print(sky.clouds.service_catalog.constants.CATALOG_SCHEMA_VERSION)')
versions=$(cat catalogs/.metadata.yml | yq -r '.version_commit | keys[]')
cd catalogs
git config --global user.name 'github-actions[bot]'
git config --global user.email 'github-actions[bot]@users.noreply.github.com'
git add .
git commit -m"[Bot] Update Lambda catalog $version (scheduled at $(date))" || { echo "No changes to commit" && exit 0; }
git commit -m"[Bot] Update Lambda catalog $versions (scheduled at $(date))" || { echo "No changes to commit" && exit 0; }
git fetch origin
git rebase origin/master
git push
9 changes: 9 additions & 0 deletions .metadata.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Mapping between version and SkyPilot commit. The CI will update the catalogs
# for all versions listed here. We keep older versions for backward
# compatibility.
# TODO(zhwu): Remove v5 support after 0.10.0 release.
version_commit:
# This commit should be fixed, as any newer commit will be fetching v6
# catalogs
v5: 1ed40e3174646ba835423b9308d8d6489f83b6bc
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just so I understand: when we update catalog fetchers in our main skypilot repo, this hash would also need to be updated?

If so, we should probably make a note of this in README.md

v6: latest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also add a quick update to README.md for this repo that v6 is now a supported catalog version? Can be another PR too.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! Updated.

12 changes: 6 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
# SkyPilot Catalogs

**Latest catalog schema version**: v5
**Latest catalog schema version**: v6

**Supported catalog schema versions**: v1, v2, v3, v4, v5
**Supported catalog schema versions**: v1, v2, v3, v4, v5, v6

**The versions that are still being updated periodically**: v5, v6

## Automatic Catalog Fetching

Expand All @@ -12,11 +14,9 @@ Catalogs are updated **every 7 hours**.



## Schema V6


## Schema V5

The catalogs for each cloud in [v5](v5) include the following files:
The catalogs for each cloud in [v6](v6) include the following files:
1. `vms.csv`: the catalog for the VMs, including the instance and the accelerators.
2. `images.csv`: the catalog for the images, which contains the mapping from the SkyPilot image tag to the image ID that can be used to find the image in the clouds.

Expand Down