Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/pinning file support #22

Merged
merged 83 commits into from
Sep 13, 2024
Merged
Show file tree
Hide file tree
Changes from 79 commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
8a8bbc6
added pinning_utils
lyon040502003 Jul 12, 2024
cd2ee23
rebuild the pinning functions to work on raw Usd instead of the UsdU…
lyon040502003 Jul 22, 2024
f8e71d9
Update client/ayon_usd/standalone/Usd/Pinning/pinning_file_helper.py
Lypsolon Jul 22, 2024
e328b12
Update client/ayon_usd/standalone/Usd/Pinning/pinning_file_helper.py
Lypsolon Jul 22, 2024
68fd888
Update client/ayon_usd/standalone/Usd/Pinning/pinning_file_helper.py
Lypsolon Jul 22, 2024
c773718
Update client/ayon_usd/standalone/Usd/Pinning/pinning_file_helper.py
Lypsolon Jul 22, 2024
ed6635b
Update client/ayon_usd/standalone/Usd/Pinning/pinning_file_helper.py
Lypsolon Jul 22, 2024
cd46c60
Update client/ayon_usd/addon.py
Lypsolon Jul 22, 2024
c803470
Update server/settings/main.py
Lypsolon Jul 22, 2024
328abc1
implemented a function that allows creating a pinning file from a sin…
lyon040502003 Jul 23, 2024
5df1a80
updated pinning fiel helper and ayon_bin_client to new version
lyon040502003 Jul 25, 2024
e434c48
solving merge conflicts
lyon040502003 Aug 12, 2024
b375494
added Pinning suport excample
lyon040502003 Aug 12, 2024
868287c
fixed the excample code and renamed foulders to be pep compatible
lyon040502003 Aug 12, 2024
69b91a4
implemented a switch that forces the studio to accept that the addon …
lyon040502003 Aug 14, 2024
3851f9f
added an pyblish plugin that will generate a pinning file for every u…
lyon040502003 Aug 15, 2024
edf79ad
added infos about keys in the pinning file to the Dev Docs
lyon040502003 Aug 19, 2024
ef5fc83
added missing doc string info for pinning file helper
lyon040502003 Aug 20, 2024
61e5ef2
Update Docs/Ayon_Docs/Admin_Docs.md
Lypsolon Aug 20, 2024
a1fbebb
Update Docs/Ayon_Docs/Admin_Docs.md
Lypsolon Aug 20, 2024
b7f7c79
updated main.py docstrings
lyon040502003 Aug 20, 2024
0daec78
changed version to create release canedate
lyon040502003 Aug 22, 2024
9831deb
added a - to the README so that i can trigger the workflow
lyon040502003 Aug 22, 2024
d8a583f
Merge branch 'develop' into feature/pinning_file_suport
antirotor Aug 22, 2024
aedc9f3
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
fd72bd6
removed strange change log looking text from admin docs
lyon040502003 Aug 22, 2024
e5f059b
get_plugin_paths used the wrong function descriptor
lyon040502003 Aug 22, 2024
a309d23
added that get_publish_plugin_paths only adds a path if the path actu…
lyon040502003 Aug 22, 2024
c89fb83
exposing the generate_pinning_file function from the pinning_file_hel…
lyon040502003 Aug 22, 2024
a853870
refactored pinning_file_helper so that every function that uses a res…
lyon040502003 Aug 22, 2024
60efbd0
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
aa9a7bf
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
d09f0fe
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
29e3e11
cleand _traverse_prims and renamed to _traverse_prim_specs
lyon040502003 Aug 22, 2024
58f7281
renamed _traverse_prim_specs to _get_prim_sepc_hierarchy_external_re…
lyon040502003 Aug 22, 2024
9a5d62e
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
1f14c06
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
53b17ba
removed excample addon
lyon040502003 Aug 22, 2024
f064933
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
24af575
Update client/ayon_usd/version.py
Lypsolon Aug 22, 2024
7674451
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
52281f8
refactored get_asset_dependencies to rename identifier_to_path_dict t…
lyon040502003 Aug 22, 2024
8c2e8bd
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
bd18328
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
e2bf871
added layer traversel deduplication to get_asset_dependencys
lyon040502003 Aug 23, 2024
8095adb
fixes for linting
lyon040502003 Aug 23, 2024
cb715d6
fixes for linting
lyon040502003 Aug 23, 2024
92f856a
added check to get_asset_dependencys to avoid recursion
lyon040502003 Aug 23, 2024
72b5e4f
updated logic to deduplicate layer taversel in get_asset_dependecys
lyon040502003 Aug 23, 2024
8151545
removed get publish plugin paths implementation as we need to do that…
lyon040502003 Aug 23, 2024
ea91dcb
removed IPluginPaths to do that in a different pr
lyon040502003 Aug 23, 2024
4846643
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 23, 2024
a08f515
implemented new version to hanlde udims from BigRoy
lyon040502003 Aug 23, 2024
386a7df
Update client/ayon_usd/addon.py
Lypsolon Aug 23, 2024
a061e48
Update Docs/Ayon_Docs/Dev_Docs.md
Lypsolon Aug 23, 2024
5f56010
removed old thinks makde _resolve_udim nicer to read
lyon040502003 Aug 23, 2024
8924de4
Update Docs/Ayon_Docs/Dev_Docs.md
Lypsolon Aug 23, 2024
3ab82b0
renamed pinning_file_helper to make it abelutly clear that you should…
lyon040502003 Aug 28, 2024
6678881
changed wrtie_pinning_file to raise errors instead of returning true …
lyon040502003 Aug 28, 2024
9b91640
updaded remove_root_from_dependecy_info docstring
lyon040502003 Aug 28, 2024
ecbc833
updates for conversations
lyon040502003 Aug 28, 2024
8263359
Update client/ayon_usd/standalone/usd/pinning/__pinning_file_generati…
Lypsolon Aug 28, 2024
a09471b
Update client/ayon_usd/standalone/usd/pinning/__pinning_file_generati…
Lypsolon Aug 28, 2024
13c0f8c
Update client/ayon_usd/standalone/usd/pinning/__pinning_file_generati…
Lypsolon Aug 28, 2024
b37afb1
Update client/ayon_usd/standalone/usd/pinning/__pinning_file_generati…
Lypsolon Aug 28, 2024
b37ad08
Update client/ayon_usd/standalone/usd/pinning/__pinning_file_generati…
Lypsolon Aug 28, 2024
43b789f
Update client/ayon_usd/standalone/usd/pinning/__pinning_file_generati…
Lypsolon Aug 28, 2024
22131f5
updateed to set
lyon040502003 Aug 28, 2024
3208649
updated functions for pr
lyon040502003 Aug 29, 2024
6fe32ac
upaded pre resolver init because one of the modification time checks …
lyon040502003 Aug 29, 2024
c269c93
Merge branch 'develop' of https://github.com/ynput/ayon-usd into feat…
BigRoy Sep 2, 2024
be375fb
Warn on layers that can't be opened but continue to process (it's val…
BigRoy Sep 3, 2024
e1e4a3b
Merge branch 'develop' into feature/pinning_file_suport
BigRoy Sep 3, 2024
fefa609
pinning file creation now allows to create pinning file for rel paths…
Sep 9, 2024
408ca51
fixes for linting
Sep 9, 2024
684deea
Cosmetics
BigRoy Sep 10, 2024
433305a
upade pinning generation for udims
Sep 11, 2024
02be71c
added a normalization pas to the generate_pinning_file function to en…
Sep 11, 2024
cf1bbf1
removed debug print from addon.py to avoid printing downloads dir
Sep 11, 2024
154eab6
removed testing file and changed version to 0.1.0-alpha
Sep 13, 2024
5e399e6
clean up
Sep 13, 2024
90a4a16
Update client/ayon_usd/standalone/usd/pinning/_pinning_file_generatio…
BigRoy Sep 13, 2024
d85ad96
Update client/ayon_usd/standalone/usd/pinning/_pinning_file_generatio…
Lypsolon Sep 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 28 additions & 19 deletions Docs/Ayon_Docs/Admin_Docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,33 +36,42 @@ distributing Usd and Ayon Libs:

## Configuration

there is a list of things that you can configure in the server settings to
There is a list of things that you can configure in the server settings to
achieve the optimal setup for your studio. In most cases you will probably not
need to touch them tho.
need to touch them though.

#### LakeFs Config

**LakeFs Settings:** `ayon+settings://ayon_usd/lakefs`\
LakeFs is the backend of our bin distribution system the addon will use the
specified server to download the resolvers and AyonUsdLibs from LakeFs.
**LakeFs Settings:**

The `ayon+settings://ayon_usd/lakefs` section configures the backend
for our bin distribution system, which uses the specified server to download
resolvers and AyonUsdLibs from LakeFs.


**LakeFs Server Uri:**
`ayon+settings://ayon_usd/lakefs/server_repo`\
this is the Uri used to host the LakeFs server. the Ynput server can be found at

`ayon+settings://ayon_usd/lakefs/server_repo`

This is the URI used to host the LakeFs server. You can find the Ynput server at
`https://lake.ayon.cloud`

**LakeFs Repository Uri:**
`ayon+settings://ayon_usd/lakefs/server_repo`\
this is a LakeFs internal link that also specifies the branch your downloading
from.\
this can be great if you want to pin your pipeline to a specific release.

**Asset Resolvers:** `ayon+settings://ayon_usd/lakefs/asset_resolvers`\
allows you to associate a specific Application name with a specific resolver.\
we always set up all the resolvers we compile but if you have special App_Names
in your Applications then you might want to add an App Alias.\
e.g. if you have hou19.5.xxx setup as a variant for Houdini you can then set
it as an alias for the Hou19.5 entry because they share the same resolver.
**LakeFs Repository URI:**

`ayon+settings://ayon_usd/lakefs/server_repo`

This is a LakeFs internal link that specifies the branch you are downloading
from. This can be useful if you want to pin your pipeline to a specific release.

**Asset Resolvers:**

`ayon+settings://ayon_usd/lakefs/asset_resolvers`

This allows you to associate a specific Application name with a specific
resolver. We always set up all the resolvers we compile, but if you have special
App_Names in your Applications, you might want to add an App Alias. For example,
if you have "hou19.5.xxx" set up as a variant for Houdini, you can then set it
as an alias for the Hou19.5 entry because they share the same resolver.

#### Usd Resolver Config

Expand Down
45 changes: 40 additions & 5 deletions Docs/Ayon_Docs/Dev_Docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,44 @@ Most important Locations.

### Pinning Support.

The Ayon Usd resolver has a feature we call pinning support. This allows storing
the current state of an Usd stage in a file to load the data quickly and without
server interaction on a Farm or any distributed system that might overwhelm or
impact the server performance.
`Standalone/Usd/pinning/pinning_file_helper.py`

The rest of this can be found in the pinning support Branch
The Ayon Usd resolver has a feature we call pinning support.\
This allows storing the current resolver state for an Usd stage in a file to
load the frozen stage without server interaction.

`generate_pinning_file`\
creates a pinning file JSON from a given USD stage

```py
def generate_pinning_file(
entry_usd: str, root_info: Dict[str, str], pinning_file_path: str
BigRoy marked this conversation as resolved.
Show resolved Hide resolved
):
```

Example Code:

```py
import os
import ayon_api
from ayon_usd.standalone.usd.pinning import pinning_file_helper


in_usd_file = "/path/to/usd/file.usd"
root_info = ayon_api.get_project_roots_for_site(os.environ.get("AYON_PROJECT_NAME"))

pinning_file_helper.generate_pinning_file(in_usd_file, root_info, "/path/to/output_file.json")
```

:::warning USD file entrypoint URI versus filepath The `in_usd_file` can be
anything that the AyonUsdResolver can resolve, including an AYON URI. However,
it is important to know that whatever you use as the input must be the input
when you load the usd.\
In other words: if you use an URI to generate the pinning file you need to open
the stage with the same URI, whereas if you use an file path you will need to do
the same.\
::: it is generally advised to use an URI as they will never have any system
specific data in them.\
The generated pinning file also has a key that will tell you what entry point
was used to generate the pinning file. You can simply read out this key and use
it to open a stage with the pinning file `ayon_pinning_data_entry_scene`.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ You can find Admin and Developer docs under Docs/Ayon_Docs/
### Clone the Repo

> **_IMPORTANT_**\
> This repository uses Git Submodules. Make sure to use the correct `git clone`\
> This repository uses Git-Submodules. Make sure to use the correct `git clone`\
> commands accordingly.\
> `git clone --recurse-submodules https://github.com/ynput/ayon-usd.git`\
> `git submodule update --init --recursive`
Expand Down
Empty file.
3 changes: 3 additions & 0 deletions client/ayon_usd/standalone/usd/pinning/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
from .__pinning_file_generation_funcs import generate_pinning_file

__all__ = ["generate_pinning_file"]
Loading