Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use IPLD bindnode and reduce boilerplate code in ingest schema #268

Merged
merged 1 commit into from
Mar 22, 2022

Conversation

masih
Copy link
Member

@masih masih commented Mar 16, 2022

Use IPLD bindnode to directly wrap and unwrap go structs in order to
reduce boilerplate code when encoding or decoding ingest types.

The codegen used before to generate IPLD schema types was also out of
date. Using bindnode has the added advantage that there is no need to
keep generated code up-to-date.

@codecov-commenter
Copy link

codecov-commenter commented Mar 16, 2022

Codecov Report

Merging #268 (a326700) into main (b4de103) will increase coverage by 15.96%.
The diff coverage is 77.57%.

Impacted file tree graph

@@             Coverage Diff             @@
##             main     #268       +/-   ##
===========================================
+ Coverage   29.34%   45.31%   +15.96%     
===========================================
  Files          84       82        -2     
  Lines       11765     6117     -5648     
===========================================
- Hits         3453     2772      -681     
+ Misses       7817     2960     -4857     
+ Partials      495      385      -110     
Impacted Files Coverage Δ
api/v0/ingest/schema/types.go 61.11% <61.11%> (ø)
api/v0/ingest/schema/schema.go 66.66% <66.66%> (ø)
internal/ingest/linksystem.go 59.01% <69.44%> (+2.04%) ⬆️
api/v0/ingest/schema/envelope.go 65.51% <75.86%> (+0.97%) ⬆️
internal/ingest/ingest.go 81.18% <100.00%> (-1.32%) ⬇️
test/typehelpers/typehelpers.go 94.58% <100.00%> (+1.41%) ⬆️
internal/registry/sequences.go 47.05% <0.00%> (-23.53%) ⬇️
internal/registry/registry.go 68.04% <0.00%> (-3.68%) ⬇️
... and 2 more

@masih masih marked this pull request as ready for review March 16, 2022 22:35
@masih masih requested review from gammazero, MarcoPolo and mvdan March 16, 2022 22:35
Copy link
Collaborator

@gammazero gammazero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all looks good, but I do not think I am qualified to review this. For example, I cannot say this change here is absolutely correct without learning more about IPLD schema generation, bindnode, etc.

@masih
Copy link
Member Author

masih commented Mar 17, 2022

Thanks for looking @gammazero. I have tagged Daniel as a review to check IPLD stuff.

Copy link
Contributor

@mvdan mvdan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM, I'd just recommend avoiding these repeated methods on types if possible, as I think using bindnode directly should be pretty straightforward.

api/v0/ingest/schema/schema.go Outdated Show resolved Hide resolved
api/v0/ingest/schema/schema.go Outdated Show resolved Hide resolved
api/v0/ingest/schema/schema.go Outdated Show resolved Hide resolved
api/v0/ingest/schema/types.go Show resolved Hide resolved

// FromNode populates the fields of this advertisement from the given node.
func (a *Advertisement) FromNode(node ipld.Node) error {
if node.Prototype() != AdvertisementPrototype {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When do you expect this to kick in? I would document it.

If it's kicking in often, then I imagine something's not right - perhaps your link loading stuff is using basicnode.Prototype.Any when following links, and then you're doing the extra work of converting the untyped node to Advertisement here. At the very least this needs a TODO to avoid the duplicate work at some point in the future.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The linksystem is passed around to other libraries like go-legs which might load things using Any and I don't have control over that unfortunately. All code in this repo, however uses appropriate prototype when loading.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fine, I would just document it then. I think @willscott, you, and I could briefly chat about this at some point, so that we can properly plumb a NodePrototypeChooser through libraries like go-legs.

api/v0/ingest/schema/types.go Outdated Show resolved Hide resolved
api/v0/ingest/schema/types.go Outdated Show resolved Hide resolved
api/v0/ingest/schema/types_test.go Outdated Show resolved Hide resolved
api/v0/ingest/schema/types_test.go Outdated Show resolved Hide resolved
}
EntryChunk struct {
Entries []multihash.Multihash
Next *ipld.Link
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this pointer to interface feels weird, because an interface can already be nil, which should be enough for optional/nullable. See ipld/go-ipld-prime#378 FYI

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great to see that issue captured 👍 In the meantime, I have to use pointer to interfaces right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup.

@masih masih force-pushed the masih/use-ipld-bindnode branch 2 times, most recently from dd46a1c to a326700 Compare March 17, 2022 12:36
@masih
Copy link
Member Author

masih commented Mar 17, 2022

@gammazero that's all Daniel's review comments addressed, and branch rebased from main.

This is ready for another look whenever you get the chance.

@masih masih requested a review from willscott March 17, 2022 15:26
masih added a commit to ipni/index-provider that referenced this pull request Mar 17, 2022
Reflect changes in storetheindex that replaces IPLD codegen schema with
bindnode structs.

See:
- ipni/storetheindex#268
Copy link
Collaborator

@MarcoPolo MarcoPolo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

over 10:1 delete to add ratio. Nice!

Use IPLD bindnode to directly wrap and unwrap go structs in order to
reduce boilerplate code when encoding or decoding ingest types.

The codegen used before to generate IPLD schema types was also out of
date. Using bindnode has the added advantage that there is no need to
keep generated code up-to-date.
@masih masih force-pushed the masih/use-ipld-bindnode branch from a326700 to d35a321 Compare March 22, 2022 10:24
@masih masih merged commit 3e82a3c into main Mar 22, 2022
@masih masih deleted the masih/use-ipld-bindnode branch March 22, 2022 10:39
masih added a commit to ipni/index-provider that referenced this pull request Mar 22, 2022
Reflect changes in storetheindex that replaces IPLD codegen schema with
bindnode structs.

See:
- ipni/storetheindex#268
masih added a commit to ipni/index-provider that referenced this pull request Mar 22, 2022
Reflect changes in storetheindex that replaces IPLD codegen schema with
bindnode structs.

See:
- ipni/storetheindex#268
masih added a commit to ipni/index-provider that referenced this pull request Mar 22, 2022
Reflect changes in storetheindex that replaces IPLD codegen schema with
bindnode structs.

See:
- ipni/storetheindex#268
masih added a commit to ipni/index-provider that referenced this pull request Mar 22, 2022
Reflect changes in storetheindex that replaces IPLD codegen schema with
bindnode structs.

See:
- ipni/storetheindex#268
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants