Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add telemetry for piece indexer #84

Merged
merged 14 commits into from
Feb 7, 2025

Conversation

NikolasHaimerl
Copy link
Contributor

@NikolasHaimerl NikolasHaimerl commented Feb 5, 2025

The piece CID indexer fetches the payload CIDs and if it can successfully retrieve a payload CID for a given (miner_id, piece_cid) combination it will then store it in the database.
To observe the piece CID indexer two metrics are highly relevant to keep track of.

  1. Add a telemetry entry for the total number of missing payloads in the database
  • The total number of payloads that are still missing from the database after the piece indexer loop has concluded. This metric should ideally have a growth that is lower than the number of deals inserted in each deal-observer round. Otherwise it would mean that the piece CID indexer cannot keep up with the number of deals being inserted into the database.
  1. Add a telemetry entry for the number of missing payloads resolved in a given indexing round
  • The number of payloads that are retrieved and updated in tha database each round of the piece CID indexer . We want to keep track of this number so we can analyze the rate at which payloads are retrieved in comparison the the number of deals that are being inserted.

@NikolasHaimerl NikolasHaimerl marked this pull request as ready for review February 5, 2025 13:01
Copy link
Member

@juliangruber juliangruber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to address #85 first

backend/bin/deal-observer-backend.js Show resolved Hide resolved
@juliangruber
Copy link
Member

@NikolasHaimerl can you please resolve merge conflicts and take a stab at unifying naming?

@juliangruber juliangruber self-requested a review February 6, 2025 06:47
Copy link
Member

@juliangruber juliangruber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there 👏

backend/bin/deal-observer-backend.js Show resolved Hide resolved
backend/bin/deal-observer-backend.js Outdated Show resolved Hide resolved
backend/lib/look-up-payload-cids.js Show resolved Hide resolved
Copy link
Member

@juliangruber juliangruber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there 👍 Just one name to update and (if I understand correctly) one follow-up issue to create

backend/bin/deal-observer-backend.js Show resolved Hide resolved
backend/bin/deal-observer-backend.js Outdated Show resolved Hide resolved
backend/bin/deal-observer-backend.js Outdated Show resolved Hide resolved
Comment on lines 130 to 133
recordTelemetry('look_up_payload_cids_stats', point => {
point.intField('total_unresolved_payload_cids', numOfUnresolvedPayloadsCids)
point.intField('payload_cids_resolved', numOfPayloadsCidsResolved)
})
Copy link
Contributor

@pyropy pyropy Feb 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could simplify all names, include the metric names, WDYT?

Suggested change
recordTelemetry('look_up_payload_cids_stats', point => {
point.intField('total_unresolved_payload_cids', numOfUnresolvedPayloadsCids)
point.intField('payload_cids_resolved', numOfPayloadsCidsResolved)
})
recordTelemetry('look_up_payload_cids_stats', point => {
point.intField('unresolved_payload_cids', numOfUnresolvedPayloadsCids)
point.intField('resolved_payload_cids, numOfPayloadsCidsResolved)
})

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

total_unresolved_payload_cids refers to the total number all of unresolved payload cids, while resolved_payload_cids only refers to those resolved in a single round. Putting the unresolved/resolved at the beginning of the variable name does make sense though.

backend/lib/look-up-payload-cids.js Show resolved Hide resolved
backend/bin/deal-observer-backend.js Outdated Show resolved Hide resolved
backend/bin/deal-observer-backend.js Outdated Show resolved Hide resolved
@NikolasHaimerl NikolasHaimerl requested a review from pyropy February 7, 2025 11:44
Copy link
Member

@juliangruber juliangruber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ship it 🥳

@NikolasHaimerl NikolasHaimerl enabled auto-merge (squash) February 7, 2025 13:06
Copy link
Contributor

@pyropy pyropy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@NikolasHaimerl NikolasHaimerl merged commit b9f9cc6 into main Feb 7, 2025
8 checks passed
@NikolasHaimerl NikolasHaimerl deleted the nhaimerl-add-telemetry-for-missing-payloads branch February 7, 2025 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants