Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/pinning file support #22

Merged
merged 83 commits into from
Sep 13, 2024
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
8a8bbc6
added pinning_utils
lyon040502003 Jul 12, 2024
cd2ee23
rebuild the pinning functions to work on raw Usd instead of the UsdU…
lyon040502003 Jul 22, 2024
f8e71d9
Update client/ayon_usd/standalone/Usd/Pinning/pinning_file_helper.py
Lypsolon Jul 22, 2024
e328b12
Update client/ayon_usd/standalone/Usd/Pinning/pinning_file_helper.py
Lypsolon Jul 22, 2024
68fd888
Update client/ayon_usd/standalone/Usd/Pinning/pinning_file_helper.py
Lypsolon Jul 22, 2024
c773718
Update client/ayon_usd/standalone/Usd/Pinning/pinning_file_helper.py
Lypsolon Jul 22, 2024
ed6635b
Update client/ayon_usd/standalone/Usd/Pinning/pinning_file_helper.py
Lypsolon Jul 22, 2024
cd46c60
Update client/ayon_usd/addon.py
Lypsolon Jul 22, 2024
c803470
Update server/settings/main.py
Lypsolon Jul 22, 2024
328abc1
implemented a function that allows creating a pinning file from a sin…
lyon040502003 Jul 23, 2024
5df1a80
updated pinning fiel helper and ayon_bin_client to new version
lyon040502003 Jul 25, 2024
e434c48
solving merge conflicts
lyon040502003 Aug 12, 2024
b375494
added Pinning suport excample
lyon040502003 Aug 12, 2024
868287c
fixed the excample code and renamed foulders to be pep compatible
lyon040502003 Aug 12, 2024
69b91a4
implemented a switch that forces the studio to accept that the addon …
lyon040502003 Aug 14, 2024
3851f9f
added an pyblish plugin that will generate a pinning file for every u…
lyon040502003 Aug 15, 2024
edf79ad
added infos about keys in the pinning file to the Dev Docs
lyon040502003 Aug 19, 2024
ef5fc83
added missing doc string info for pinning file helper
lyon040502003 Aug 20, 2024
61e5ef2
Update Docs/Ayon_Docs/Admin_Docs.md
Lypsolon Aug 20, 2024
a1fbebb
Update Docs/Ayon_Docs/Admin_Docs.md
Lypsolon Aug 20, 2024
b7f7c79
updated main.py docstrings
lyon040502003 Aug 20, 2024
0daec78
changed version to create release canedate
lyon040502003 Aug 22, 2024
9831deb
added a - to the README so that i can trigger the workflow
lyon040502003 Aug 22, 2024
d8a583f
Merge branch 'develop' into feature/pinning_file_suport
antirotor Aug 22, 2024
aedc9f3
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
fd72bd6
removed strange change log looking text from admin docs
lyon040502003 Aug 22, 2024
e5f059b
get_plugin_paths used the wrong function descriptor
lyon040502003 Aug 22, 2024
a309d23
added that get_publish_plugin_paths only adds a path if the path actu…
lyon040502003 Aug 22, 2024
c89fb83
exposing the generate_pinning_file function from the pinning_file_hel…
lyon040502003 Aug 22, 2024
a853870
refactored pinning_file_helper so that every function that uses a res…
lyon040502003 Aug 22, 2024
60efbd0
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
aa9a7bf
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
d09f0fe
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
29e3e11
cleand _traverse_prims and renamed to _traverse_prim_specs
lyon040502003 Aug 22, 2024
58f7281
renamed _traverse_prim_specs to _get_prim_sepc_hierarchy_external_re…
lyon040502003 Aug 22, 2024
9a5d62e
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
1f14c06
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
53b17ba
removed excample addon
lyon040502003 Aug 22, 2024
f064933
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
24af575
Update client/ayon_usd/version.py
Lypsolon Aug 22, 2024
7674451
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
52281f8
refactored get_asset_dependencies to rename identifier_to_path_dict t…
lyon040502003 Aug 22, 2024
8c2e8bd
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
bd18328
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 22, 2024
e2bf871
added layer traversel deduplication to get_asset_dependencys
lyon040502003 Aug 23, 2024
8095adb
fixes for linting
lyon040502003 Aug 23, 2024
cb715d6
fixes for linting
lyon040502003 Aug 23, 2024
92f856a
added check to get_asset_dependencys to avoid recursion
lyon040502003 Aug 23, 2024
72b5e4f
updated logic to deduplicate layer taversel in get_asset_dependecys
lyon040502003 Aug 23, 2024
8151545
removed get publish plugin paths implementation as we need to do that…
lyon040502003 Aug 23, 2024
ea91dcb
removed IPluginPaths to do that in a different pr
lyon040502003 Aug 23, 2024
4846643
Update client/ayon_usd/standalone/usd/pinning/pinning_file_helper.py
Lypsolon Aug 23, 2024
a08f515
implemented new version to hanlde udims from BigRoy
lyon040502003 Aug 23, 2024
386a7df
Update client/ayon_usd/addon.py
Lypsolon Aug 23, 2024
a061e48
Update Docs/Ayon_Docs/Dev_Docs.md
Lypsolon Aug 23, 2024
5f56010
removed old thinks makde _resolve_udim nicer to read
lyon040502003 Aug 23, 2024
8924de4
Update Docs/Ayon_Docs/Dev_Docs.md
Lypsolon Aug 23, 2024
3ab82b0
renamed pinning_file_helper to make it abelutly clear that you should…
lyon040502003 Aug 28, 2024
6678881
changed wrtie_pinning_file to raise errors instead of returning true …
lyon040502003 Aug 28, 2024
9b91640
updaded remove_root_from_dependecy_info docstring
lyon040502003 Aug 28, 2024
ecbc833
updates for conversations
lyon040502003 Aug 28, 2024
8263359
Update client/ayon_usd/standalone/usd/pinning/__pinning_file_generati…
Lypsolon Aug 28, 2024
a09471b
Update client/ayon_usd/standalone/usd/pinning/__pinning_file_generati…
Lypsolon Aug 28, 2024
13c0f8c
Update client/ayon_usd/standalone/usd/pinning/__pinning_file_generati…
Lypsolon Aug 28, 2024
b37afb1
Update client/ayon_usd/standalone/usd/pinning/__pinning_file_generati…
Lypsolon Aug 28, 2024
b37ad08
Update client/ayon_usd/standalone/usd/pinning/__pinning_file_generati…
Lypsolon Aug 28, 2024
43b789f
Update client/ayon_usd/standalone/usd/pinning/__pinning_file_generati…
Lypsolon Aug 28, 2024
22131f5
updateed to set
lyon040502003 Aug 28, 2024
3208649
updated functions for pr
lyon040502003 Aug 29, 2024
6fe32ac
upaded pre resolver init because one of the modification time checks …
lyon040502003 Aug 29, 2024
c269c93
Merge branch 'develop' of https://github.com/ynput/ayon-usd into feat…
BigRoy Sep 2, 2024
be375fb
Warn on layers that can't be opened but continue to process (it's val…
BigRoy Sep 3, 2024
e1e4a3b
Merge branch 'develop' into feature/pinning_file_suport
BigRoy Sep 3, 2024
fefa609
pinning file creation now allows to create pinning file for rel paths…
Sep 9, 2024
408ca51
fixes for linting
Sep 9, 2024
684deea
Cosmetics
BigRoy Sep 10, 2024
433305a
upade pinning generation for udims
Sep 11, 2024
02be71c
added a normalization pas to the generate_pinning_file function to en…
Sep 11, 2024
cf1bbf1
removed debug print from addon.py to avoid printing downloads dir
Sep 11, 2024
154eab6
removed testing file and changed version to 0.1.0-alpha
Sep 13, 2024
5e399e6
clean up
Sep 13, 2024
90a4a16
Update client/ayon_usd/standalone/usd/pinning/_pinning_file_generatio…
BigRoy Sep 13, 2024
d85ad96
Update client/ayon_usd/standalone/usd/pinning/_pinning_file_generatio…
Lypsolon Sep 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 39 additions & 5 deletions Docs/Ayon_Docs/Dev_Docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,43 @@ Most important Locations.

### Pinning Support.

The Ayon Usd resolver has a feature we call pinning support. This allows storing
the current state of an Usd stage in a file to load the data quickly and without
server interaction on a Farm or any distributed system that might overwhelm or
impact the server performance.
`Standalone/Usd/pinning/pinning_file_helper.py`

The rest of this can be found in the pinning support Branch
The Ayon Usd resolver has a feature we call pinning support.\
This allows storing the current resolver state for an Usd stage in a file to
load the frozen stage without server interaction.

`generate_pinning_file`\
creates a pinning file JSON from a given USD stage

```py
def generate_pinning_file(
entry_usd: str, root_info: Dict[str, str], pinning_file_path: str
BigRoy marked this conversation as resolved.
Show resolved Hide resolved
):
```

Example Code:

```py
import os
import ayon_api
from ayon_usd.standalone.usd.pinning import pinning_file_helper


in_usd_file = "/path/to/usd/file.usd"
root_info = ayon_api.get_project_roots_for_site(os.environ.get("AYON_PROJECT_NAME"))

pinning_file_helper.generate_pinning_file(in_usd_file, root_info, "/path/to/output_file.json")
```

`in_usd_file` can be everything that the AyonUsdResolver can resolve including
an URI its just important to know that what ever you use as the input must be
the input when you load the usd.\
in other words: if you use an uri to generate the pinning file you need to open
the stage with the same uri, if you use an file path you will need to do the
same.\
Lypsolon marked this conversation as resolved.
Show resolved Hide resolved
it is generally advised to use an URI as they will never have any system
specific data in them.\
The generated pinning file also has a key that will tell you what entry point as
used to generate the pinning file. you can simply read out this key and use it
to open a stage with the pinning file `ayon_pinning_data_entry_sceene`
Lypsolon marked this conversation as resolved.
Show resolved Hide resolved
25 changes: 23 additions & 2 deletions client/ayon_usd/addon.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
import os
from datetime import datetime, timezone

from ayon_core.addon import AYONAddon, ITrayAddon
from ayon_core.addon import AYONAddon, ITrayAddon, IPluginPaths

from ayon_core import style

Expand All @@ -14,8 +14,10 @@
from .ayon_bin_client.ayon_bin_distro.work_handler import worker
from .ayon_bin_client.ayon_bin_distro.util import zip

USD_ADDON_ROOT = os.path.dirname(os.path.abspath(__file__))

class USDAddon(AYONAddon, ITrayAddon):

class USDAddon(AYONAddon, ITrayAddon, IPluginPaths):
Lypsolon marked this conversation as resolved.
Show resolved Hide resolved
"""Addon to add USD Support to AYON.

Addon can also skip distribution of binaries from server and can
Expand All @@ -34,6 +36,10 @@ def tray_init(self):

def initialize(self, module_settings):
"""Initialize USD Addon."""
if not module_settings["ayon_usd"]["allow_addon_start"]:
raise SystemError(
"The experimental AyonUsd addon is currently activated, but you haven't yet acknowledged the user agreement indicating your understanding that this feature is experimental. Please go to the Studio settings and check the agreement checkbox."
)
self.enabled = True
self._download_window = None

Expand Down Expand Up @@ -127,3 +133,18 @@ def tray_menu(self, tray_menu):
def get_launch_hook_paths(self):
"""Get paths to launch hooks."""
return [os.path.join(config.USD_ADDON_DIR, "hooks")]

def get_plugin_paths(self):
"""Deadline plugin paths."""
Lypsolon marked this conversation as resolved.
Show resolved Hide resolved
# Note: We are not returning `publish` key because we have overridden
# `get_publish_plugin_paths` to return paths host-specific. However,
# `get_plugin_paths` still needs to be implemented because it's
# abstract on the parent class
return {}

def get_publish_plugin_paths(self, host_name=None):
publish_dir = os.path.join(USD_ADDON_ROOT, "plugins", "publish")
paths = [os.path.join(publish_dir, "global")]
if host_name:
paths.append(os.path.join(publish_dir, host_name))
Lypsolon marked this conversation as resolved.
Show resolved Hide resolved
return paths
BigRoy marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
"""
Empty Pyblish plugin that used for inspection.

Note:
PLugin's order is an int constant.
Current Orders:
pyblish.api.CollectorOrder == 0
pyblish.api.ValidatorOrder == 1
pyblish.api.ExtractorOrder == 2
pyblish.api.IntegratorOrder == 3


Sometimes, you can add some number to the order
to adjust its order.
You may want to adjust the order to make it run
after a particular plugin.
"""

Lypsolon marked this conversation as resolved.
Show resolved Hide resolved
import os
import pyblish.api
import ayon_api
from ayon_core.pipeline.publish import KnownPublishError
from ayon_usd.standalone.usd.pinning import pinning_file_helper


root_info = ayon_api.get_project_roots_for_site(os.environ.get("AYON_PROJECT_NAME"))
Lypsolon marked this conversation as resolved.
Show resolved Hide resolved


class Dumb(pyblish.api.InstancePlugin):
"""Dumb"""

order = pyblish.api.IntegratorOrder + 0.3
label = "Dumb"
Lypsolon marked this conversation as resolved.
Show resolved Hide resolved
families = ["usdrop"]
hosts = ["houdini"]

enabled = True

def process(self, instance):

publish_dir = instance.data["publishDir"]
publish_usd_file = instance.data["representations"][0]["published_path"]
pinning_file_path = os.path.join(
publish_dir, publish_usd_file.split(".")[0] + "_pinning_file.json"
)
if not os.path.exists(publish_usd_file):
self.log.error(f"Usd file missing {publish_usd_file}")
raise KnownPublishError(f"Usd File Missing {publish_usd_file}")

pinning_file_helper.generate_pinning_file(
BigRoy marked this conversation as resolved.
Show resolved Hide resolved
publish_usd_file, root_info, pinning_file_path
)
self.log.debug(
f"Saving Pinning File for Usd Render Locatoin:{pinning_file_path}, for usd file {publish_usd_file}"
Lypsolon marked this conversation as resolved.
Show resolved Hide resolved
Lypsolon marked this conversation as resolved.
Show resolved Hide resolved
)
Empty file.
Empty file.
Loading