Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getter Docstrings #1185

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open

Getter Docstrings #1185

wants to merge 12 commits into from

Conversation

pauladkisson
Copy link
Member

@pauladkisson pauladkisson commented Jan 24, 2025

Added Returns section to all getter docstrings.

Fixes #1097

Cline (Claude 3.5 Sonnet) helped. It was so close to just getting it right after the first couple attempts, but then took a lot of coaching to get those last few...

Copy link

codecov bot commented Jan 25, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.58%. Comparing base (96dfdff) to head (857436a).
Report is 26 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1185      +/-   ##
==========================================
- Coverage   90.69%   89.58%   -1.11%     
==========================================
  Files         129      129              
  Lines        8189     8357     +168     
==========================================
+ Hits         7427     7487      +60     
- Misses        762      870     +108     
Flag Coverage Δ
unittests 89.58% <100.00%> (-1.11%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/neuroconv/basedatainterface.py 80.21% <ø> (-15.13%) ⬇️
src/neuroconv/baseextractorinterface.py 100.00% <ø> (ø)
...oconv/datainterfaces/behavior/video/video_utils.py 97.14% <ø> (ø)
...atainterfaces/behavior/video/videodatainterface.py 93.54% <ø> (ø)
...roconv/datainterfaces/ecephys/axona/axona_utils.py 57.94% <ø> (ø)
...erfaces/ecephys/baserecordingextractorinterface.py 93.87% <ø> (+0.06%) ⬆️
...nterfaces/ecephys/basesortingextractorinterface.py 80.19% <ø> (ø)
...ainterfaces/ecephys/neuroscope/neuroscope_utils.py 100.00% <ø> (ø)
...aces/ecephys/neuroscope/neuroscopedatainterface.py 93.65% <ø> (-2.39%) ⬇️
.../ecephys/openephys/openephysbinarydatainterface.py 97.29% <ø> (ø)
... and 29 more

... and 10 files with indirect coverage changes

@pauladkisson
Copy link
Member Author

pauladkisson commented Jan 28, 2025

After the meeting today, I think this kind of task would be a really good benchmark for AI agents / LLMs / prompts.

Some quantifiable questions:

  • how many true positive? (docstrings that needed to be edited that were CORRECTLY edited)
  • how many false positives? (docstrings that DIDNT need to be edited that were INCORRECTLY edited)
  • how many false negatives? (docstrings that needed to be edited that were MISSED or edited INCORRECTLY)
  • how many attempts before it completes the task? (In this case I had to re-prompt cline with different instructions 7 times until it finally got all of the docstrings I wanted)
  • Total cost
  • Total runtime

With that in mind, it might be worthwhile to branch off from neuroconv with a dedicated ai_docstrings_benchmark branch before merging these changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Documentation]: Should we add Returns section to the docstrings of our get methods (get_metadata, etc.)
1 participant