Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically resolve and index grafting dependencies #439

Open
chriswessels opened this issue Jul 20, 2022 · 10 comments · May be fixed by #481
Open

Automatically resolve and index grafting dependencies #439

chriswessels opened this issue Jul 20, 2022 · 10 comments · May be fixed by #481
Assignees
Labels

Comments

@chriswessels
Copy link
Member

When trying to index a subgraph that has a graft base, unless the base is already syncing, indexer-agent shows this error:

subgraph validation error: [the graft base is invalid: deployment not found: QmdSqd9g4VbWMJ22HBfLZ3CDC8DL95yWPompNeyER12cL1]"

graph-node:

Jul 20 02:38:17.898 ERRO subgraph_deploy failed, params: SubgraphDeployParams { name: SubgraphName("indexer-agent/mVwGejjBjE"), ipfs_hash: DeploymentHash("QmQ7xAxvwPhP4ZqiNZBscVuu5ZmfD9By1i3YmVwGejjBjE"), node_id: Some(NodeId("index-node-0")), debug_fork: None }, error: ManifestValidationError([GraftBaseInvalid("deployment not found: QmdSqd9g4VbWMJ22HBfLZ3CDC8DL95yWPompNeyER12cL1")]), component: JsonRpcServer

The agent could automatically resolve any grafting dependencies and ensure that those are indexing too. This is currently a manual process.

@fordN fordN added enhancement New feature or request indexer-agent labels Jul 27, 2022
@fordN
Copy link
Contributor

fordN commented Jul 27, 2022

Agreed, it'd be worth adding an auto-graft feature to the indexer-agent which handles the entire process of deploying dependency subgraphs, pausing those once they reach the required block, and then deploying the grafted subgraph.

Regarding UX: do you think indexers would like control over this feature ? In particularly they may want to protect themselves from deep recursion of subgraph deployment dependencies? I'm thinking we could provide either:

  • a boolean flag to tell the agent whether to automatically resolve grafting dependencies (--auto-graft) or
  • a numeric parameter to control how deep of a grafting dependency to auto resolve; can be set to 0 to turn off auto-graft (--auto-graft-resolver-depth).

@fordN fordN added this to Indexer Jul 27, 2022
@fordN fordN moved this from 🗃️ Inbox to 🙏 Feature Requests in Indexer Jul 27, 2022
@fordN fordN moved this to 🗃️ Inbox in Indexer Jul 27, 2022
@chriswessels
Copy link
Member Author

Makes total sense to me. The whole grafting tree could be resolved (and potentially invalidated against the max resolver depth) before deploying anything to graph-node too :)

@fordN fordN moved this from 🙏 Feature Requests to 📒 Todo in Indexer Aug 2, 2022
@fordN fordN moved this from 📒 Todo to 🙏 Feature Requests in Indexer Aug 4, 2022
@hopeyen hopeyen self-assigned this Aug 22, 2022
@hopeyen hopeyen moved this from 🙏 Feature Requests to 📒 Todo in Indexer Aug 22, 2022
@jian-guo-s
Copy link

尝试索引具有移植基的子图时,除非基已经同步,否则 indexer-agent 会显示此错误:

subgraph validation error: [the graft base is invalid: deployment not found: QmdSqd9g4VbWMJ22HBfLZ3CDC8DL95yWPompNeyER12cL1]"

图节点:

Jul 20 02:38:17.898 ERRO subgraph_deploy failed, params: SubgraphDeployParams { name: SubgraphName("indexer-agent/mVwGejjBjE"), ipfs_hash: DeploymentHash("QmQ7xAxvwPhP4ZqiNZBscVuu5ZmfD9By1i3YmVwGejjBjE"), node_id: Some(NodeId("index-node-0")), debug_fork: None }, error: ManifestValidationError([GraftBaseInvalid("deployment not found: QmdSqd9g4VbWMJ22HBfLZ3CDC8DL95yWPompNeyER12cL1")]), component: JsonRpcServer

代理可以自动解决任何嫁接依赖关系并确保它们也被索引。目前这是一个手动过程。

When trying to index a subgraph that has a graft base, unless the base is already syncing, indexer-agent shows this error:

subgraph validation error: [the graft base is invalid: deployment not found: QmdSqd9g4VbWMJ22HBfLZ3CDC8DL95yWPompNeyER12cL1]"

graph-node:

Jul 20 02:38:17.898 ERRO subgraph_deploy failed, params: SubgraphDeployParams { name: SubgraphName("indexer-agent/mVwGejjBjE"), ipfs_hash: DeploymentHash("QmQ7xAxvwPhP4ZqiNZBscVuu5ZmfD9By1i3YmVwGejjBjE"), node_id: Some(NodeId("index-node-0")), debug_fork: None }, error: ManifestValidationError([GraftBaseInvalid("deployment not found: QmdSqd9g4VbWMJ22HBfLZ3CDC8DL95yWPompNeyER12cL1")]), component: JsonRpcServer

The agent could automatically resolve any grafting dependencies and ensure that those are indexing too. This is currently a manual process.

How to handle this problem manually, thank you.

@hopeyen
Copy link
Contributor

hopeyen commented Sep 1, 2022

Hello @jian-guo-s ! The manual solution is to deploy and sync the graft base deployment upto the graft block before deploying the intended deployment.

In your case, deployment QmQ7xA.... graft onto the base deployment QmdSqd... at block 14347193. I would suggest for you to offchain sync QmdSqd..., check the indexing status until it syncs pass 14347193, and then try again with QmQ7xA...

@hopeyen hopeyen moved this from 📒 Todo to 🏗️ In Progress in Indexer Sep 12, 2022
@hopeyen hopeyen moved this from 🏗️ In Progress to 🚩 In Review in Indexer Oct 7, 2022
@fordN fordN moved this from 🚩 In Review to 🏗️ In Progress in Indexer Mar 27, 2023
@fordN fordN added the p1 label Aug 8, 2023
@cryptovestor21
Copy link

I've just started looking at how to manage these graft dependencies and would like to +1 and bump this as a feature request.

After a discussion with ellipfra/Marc-Andre, it would also be really useful to have an option that would also delete the base once the graft copy is completed

@matthewdarwin
Copy link

If this is implemented then we don't need the note here: graphprotocol/docs#501

@datanexus-vincent
Copy link

+1 to this feature. I also like the idea of it being a number you set for graft depth.

It would also be useful if graph-node automatically checked something like 2n or 3n grafts deep so that it could report the number of required grafts in the deployment error (or that it exhausted the max search).

And like @cryptovestor21 mentioned, an option to automatically delete graft bases would be much appreciated.

@alex-pakalniskis alex-pakalniskis added this to the Indexer Experience milestone Sep 26, 2023
@nahueloyha
Copy link

This feature would be awesome indeed, just had a case where a subgraph depended on other 4 grafts, making it quite time consuming to resolve and index all of them manually.

@alex-pakalniskis alex-pakalniskis moved this from 🏗️ In Progress to 🛑 Blocked in Indexer Nov 20, 2023
@alex-pakalniskis
Copy link
Contributor

User request for this feature during IOH June 25, 2024

image

@alex-pakalniskis
Copy link
Contributor

Paraphrased feedback on preferred UX

Vince | Nodeify, InfraDAO, & Pinax

  • Software should just handle the grafting without user specifying required grafts, i.e. nothing should need to be specified differently than a subgraph without grafts
  • Wants the ability to bulk remove or delete unneeded graft dependency automatically with some override ability (suggested a new CLI option for graphman commands)

Pseudo code UX sketches

graphman remove --grafts Qmwhatever where by default, "orphaned grafts" not depended on by any other subgraph are affected.

graphman remove --grafts Qmwhatever --exclude Qmspecificgraft where the Indexer passes a graft to exclude from being removed/deleted, i.e. needs the graft for another operation later today

@fordN fordN moved this from 🛑 Blocked to Up next in Indexer Jul 8, 2024
@dwerner dwerner moved this from ⌛ Up next to 🏗️ In Progress in Indexer Aug 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: 🏗️ In Progress
Development

Successfully merging a pull request may close this issue.

10 participants