Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix robot-name tests #403

Merged
merged 2 commits into from
Sep 9, 2024
Merged

Fix robot-name tests #403

merged 2 commits into from
Sep 9, 2024

Conversation

timgott
Copy link
Contributor

@timgott timgott commented Sep 8, 2024

The regex also matched on sub strings.
The other tests did not do anything because they only returned a boolean instead of using check from rackunit.

@BNAndras
Copy link
Member

BNAndras commented Sep 8, 2024

Thanks, I’ll take a look tomorrow.

@BNAndras BNAndras added x:module/practice-exercise Work on Practice Exercises x:rep/small Small amount of reputation labels Sep 8, 2024
@BNAndras BNAndras self-requested a review September 8, 2024 16:20
@blakelewis
Copy link
Contributor

Good catch! I guess that the test for unique names was doubly broken, since besides not doing a proper check, it was comparing with the wrong value. Generating 169,000 names seems like a lot of work compared to what most tests do, but we seem to be staying within our budget. The "birthday paradox" argument suggests that something in the neighborhood of sqrt(max-names) (like 1000) ought to be sufficient for this kind of test.

@BNAndras
Copy link
Member

BNAndras commented Sep 9, 2024

The exercise completion rate on this track is a little low for my taste at 63.6%, but there is an average of 2.8 attempts for every started solution, which is pretty solid. By comparison, Clojure is at 83.3% completion rate and an average of 4.5 attempts. Common Lisp is at 86.6% completion with an average of 3.2 attempts.

Common Lisp doesn't implement this test, but Clojure generates 4000 names. Java uses 5,000 and Vim script uses 10,000 names. So an argument could be made that 169k names Is a lot by comparison. I don't believe though we're causing the test runner to time out more than usual. If the number of names was a bottleneck, I'd expect to see more attempts on average. My reading of Blake's post is that they're not overly concerned at this point so I think it's fine to leave the number alone for now.

At this point, the test runner will need to rerun all 108 completed solutions to make sure they pass the updated suite. That means there's a financial impact (not a massive one but one we should be aware of) to merging this in, but it's worthwhile because the tests weren't doing what they were supposed to be.

@timgott
Copy link
Contributor Author

timgott commented Sep 9, 2024

Birthday paradox is a good point. I didn't measure exactly but the test took roughly 2-3 seconds on my laptop for my solution (and I did what all the published solutions did, with a hash set and retry on collision). With 5000 checks it would fail to detect a collision with 0.00000094% probability according to online calculator (https://instacalc.com/28845). I will change it quickly.

@BNAndras BNAndras merged commit d19fedf into exercism:main Sep 9, 2024
7 checks passed
@BNAndras
Copy link
Member

BNAndras commented Sep 9, 2024

Thanks for the fix. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
x:module/practice-exercise Work on Practice Exercises x:rep/small Small amount of reputation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants