nosub -p download only grabs first hit in batch #1
Comments
|
It looks is because when hits are still pending, this condition never returns. This was somewhat unexpected behavior (i.e. I expected to be able to pull down partial results while waiting for others to finish). |
|
Thanks for letting me know -- this is definitely unexpected behavior and should be fixed. I'm not 100% certain of your diagnosis -- that bit of code you highlighted handles paginated downloads for a single HIT. At any rate, I'm in a conference push right now, so won't be able to get to this for about a week or so. |
|
no worries -- good luck with nips 💯 |
longouyang
changed the title
|
hi long -- any progress on this? we have a few people joining us over the summer who will be starting to run experiments soon and it'd be great if this were taken care of (hopefully won't require too big a change once the issue is diagnosed) |
|
i was on a computer-less trip for a week and then got sidelined by a cold. i think i should be able to take a look at it this week though. |
|
for the example that you posted, what does your directory structure look like? |
|
also, do you have a currently running HIT that can reproduce this problem? |
|
Yes, it's not specific to any HIT -- any time you turn batch mode on, this happens. |
|
and there's no special directory structure, I'm creating an empty directory and typing |
|
Apologies for the long delay. In the months since we last discussed this, I've picked this back up a couple times but never managed to reproduce this behavior. And debugging is hard because there's a server side component I can't control and testing in production would cost money. The next time you encounter this, if it's not urgent, could you expire the HIT, and send me your nosub + auth files for debugging? |
|
Bumping this. Anything we can do to help diagnose the issue? |
|
If you encounter this bug, can you expire the HIT and send me your nosub + auth files for debugging? |
|
Hi Long, Thanks for making nosub! I'm having the same issue. Where should I send you my files? Thanks, Edit: Just sent you the files to your Gmail address! |
|
Update, for posterity: I was trying to download HITs from an experiment that was split into six batches. nosub was tracking the status of all six: But it would only download HITs from the first two batches: I sent Long my experiment files and authentication files, and he was unable to reproduce the error. I tried it again myself, and I was able to download files from all six batches: I have no idea why the problem fixed itself. Maybe it was a fluke? I'll let you know if I run into this again! |
|
@nataliavelez : I assume the problem was fixed after the final participant finished. it seems to be 'blocked' by whichever batch isn't yet complete! the problem specifically happens when some HITs are still pending, which makes it difficult to send @longouyang the files needed to reproduce it in time, i.e. before they're no longer pending. For example, I'm running into it now with this compensation HIT: It hangs without being able to download the completed HIT, and note that the HIT ID that blocks it is the one with the pending HIT... |
|
Hi Robert—Aaaaahhhh, so that's it! You're right, the problem does fix
itself when there are no HITs pending. This is so helpful!
…On Fri, Sep 27, 2019 at 10:24 PM Robert X.D. Hawkins < ***@***.***> wrote:
@nataliavelez <https://github.com/nataliavelez> : I assume the problem
was fixed after the final participant finished. it seems to be 'blocked' by
whichever batch isn't yet complete! the problem specifically happens when
some HITs are still pending, which makes it difficult to send @longouyang
<https://github.com/longouyang> the files needed to reproduce it in time,
i.e. before they're no longer pending. For example, I'm running into it now
with this compensation HIT:
➜ nosub -p status
Running on production
ID Created Expiration Assignments NumPending NumAvailable NumCompleted
------------------------------ -------------------- -------------------- ----------- ---------- ------------ ------------
3KA7IJSNW656VL2EL0DBAVQVEO6BP5 9/27/2019 3:10:36 AM 10/1/2019 3:10:36 AM 9 0 9 0
3511RHPADVE3K745P56UTTMFR08LRU 9/27/2019 3:10:36 AM 10/1/2019 3:10:36 AM 7 0 6 1
3Z3R5YC0P3NU0U717J8RYVIDIPJFTW 9/27/2019 3:07:09 AM 9/29/2019 3:07:09 AM 9 1 8 0
Total available: 25
Total completed: 1
➜ compensation_rxdh git:(master) ✗ nosub -p download
Running on production
Getting status of HIT 3Z3R5YC0P3NU0U717J8RYVIDIPJFTW
We have 0/1 assignments
It hangs without being able to download the completed HIT, and note that
the HIT ID that blocks it is the one with the pending HIT...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1?email_source=notifications&email_token=ACFEPEBRQSGOUKIAUI6O6VTQL3TBXA5CNFSM4E7K3622YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD72RK6Q#issuecomment-536155514>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACFEPEHKODCKT6JWQZKWK6TQL3TBXANCNFSM4E7K362Q>
.
|
|
Hurrah for generalizing from multiple examples! (I couldn't see what my cases had in common until I saw yours!) |
|
Just chiming in here to signal that I'm keeping tabs on this. I'm trying to reconcile this hypothesis with some earlier detective work that Natalia and I had done which seems inconsistent with the explanation (but maybe there are multiple issues?). Summarizing that investigation: she had sent me a HIT which was having the issue but which we had expired for debugging purposes. I didn't find any bugs but eventually (after a few days) downloading the results Just Worked, even though no new completions occurred. My current thinking is that Robert's explanation is likely right but there's other weirdness going on as well. I'll devote some time to more investigation this week. |
|
hmmm, interesting! one possible reconciliation (still just brainstorming) is that participants can hold on to an assignment as 'pending' for some time after the HIT is officially expired by the experimenter, with that additional time set by the assignments duration. Not sure what the assignment duration was, but maybe it started working after they eventually returned it (thus why no further completions were recorded; it moved out of the 'pending' column to the 'available' column?) It's also likely there's just totally other weirdness explaining it! :) |
Thanks for creating this! We'd made a custom shim around cosub to handle batching and bonuses and such but this is much nicer. Just wanted to report an issue. I launched 20 assignments in batch mode. Running
cosub-p statusreports that there are 3 hits in the batch with results, but when I runcosub -p downloadit looks like it only pulls results from the first in the list instead of looping through all of them (see screenshot below).The text was updated successfully, but these errors were encountered: