From e11878e2658f2353d1174ec5067b9b07059fc3c1 Mon Sep 17 00:00:00 2001 From: bumblefudge Date: Sat, 30 Dec 2023 17:19:36 -0800 Subject: [PATCH] add meeting 10-18-23 of testingTF --- 2023-10-18-testingTF/readme.md | 69 ++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 2023-10-18-testingTF/readme.md diff --git a/2023-10-18-testingTF/readme.md b/2023-10-18-testingTF/readme.md new file mode 100644 index 0000000..eb5c10e --- /dev/null +++ b/2023-10-18-testingTF/readme.md @@ -0,0 +1,69 @@ +# 18 Oct 2023 - Helge, Bengo - Test Case Design & Documentation + +## Resolutions + +None taken + +## Minutes + +- Attendance (optional/honor-system) + - @bumblefudge +- Links + - [Agenda email](https://lists.w3.org/Archives/Public/public-swicg/2023Oct/0045.html) + - Helge's [Fediverse Testing repo](https://helge.codeberg.page/funfedidev/) + - example [test case](https://codeberg.org/helge/fediverse-features/src/branch/main/w3c/ap_follow.feature) + - [slides](https://bovine.social/assets/presentation.pdf) + - Bengo's test case and YAML-ized behavior [repo](https://git.coop/socialweb.coop/activitypub/behaviors/) + - [Which ActivityPub Server Requirements are easy to test?](https://bengo.is/blogging/easy-to-test-activitypub-requirements/) + - [Use Test Specifications to Improve Testability of ActivityPub Requirements](https://bengo.is/blogging/use-test-specifications-to-improve-testability-of-activitypub-requirements/) + - [Introducing ActivityPub Requirements Cohort 1](https://bengo.is/blogging/introducing-activitypub-requirement-testing-cohort-1/) +- Fediverse Features- Presentation by @Helge + - Context - different layers of testing (slide 1) + - Gherkin as example of situational testing + - Business case (UI-level) example, truly "BDD" (behavior-driven) + - less-obvious usage: wrapping a Unit test in a gherkin feature (Use case data slide) + - bob: are you auto-generating code or hand-writing it based on these test cases? + - Helge: there are tools for that (behave for python, for ex.), but usually worth hand-editing even if you generate code automatically + - Helge: ex unit test code corresponding to this unit-test feature is at codeberg.org/bovine/bovine/.../fedi_follow.py + - bengo: would it help to have a stable identifier for addressing the test vector or test vector template in cases where it's generated at test runtime? + - bengo: could gherkin @ decorators be used to separately-maintained vectorset? + - helge: @ decorators are like `#`s or `//`s for remarks, + - for example ![screengrab showing decorator usage for remarks](https://hedgedoc.socialweb.coop/uploads/90d92599-322d-4bf9-9dff-894cb1f270ad.png) + - helge: I would say that gherkin tooling is pretty robust and well-maintained, so we should could look around for annotation tooling? + - bengo: what's the best spec to use for gherkin syntax itself? + - helge: the syntax is so simple, there's not much to specify + - bengo: I found several, tho; should we just assume this is the canonical one? + - helge: one feature I found is "data table" + - Features wrapup: + - Bengo: would work for the license? + - Bengo: I'd like multiple drivers, partic a browser-driven one? is there a way to make multiple bindings verifiable? + - maybe parameterize waits and sleeps? + - Bengo: IMHO parameterizing would be a stitch in time for sharing across diff test environments + - Peter Zingg (in chat): [Example of browser testing with Gherkin](https://www.lambdatest.com/blog/behaviour-driven-development-by-selenium-testing-with-gherkin/) + - bengo (in chat): +1 I've done it with cucumber-js to mixed results +- Fun FediDev - Harness for dockerized servers - Presentation by @Helge + - since so many servers are already dockerized, I made a harness for spawning some locally, creating accounts on them all, and having them communicate between containers on HTTP (HTTPSigs optional) + - one glitch: doesn't currently support updates to an actor's signing key (PRs welcome) + - scribe aside: is this a job for the data portability task force? at least use-cases would help here for cross-referencing/checking against + - support table example + - bengo: just by object type? helge: none yet, but many planned + - (screenshares already-live display of test vectors, from "example" column of today's displaying support table) +- Biographical aside - actually started building htis harness to test hashtag normalization for a FEP +- Bengo's Presentation + - [repo](https://git.coop/socialweb.coop/activitypub/behaviors/) autopublishes to a [web preview](https://socialweb.coop/activitypub/behaviors/) and NPM + - returns mime type JSON so you can pipe to jq, etc + - can curl from socialweb.coop/ and get the whole set + - easy/medium/hard/nearly-impossible scheme + - would rough consensus on "easy" set unblock cooperation? + - [Which ActivityPub Server Requirements are easy to test?](https://bengo.is/blogging/easy-to-test-activitypub-requirements/) + - sorting behaviors by sub-resource required: object (in general), actor, inbox, outbox, followers collection, liked collection, likes collection, and/or shares collection + - Helge (in chat): Hell: Test cases that require THREE Servers, such [as this example](https://media.hachyderm.io/media_attachments/files/111/104/507/048/912/919/original/3982f5534dac30f0.png) + - CTA - do people have more or less easies? +- Discussion/Q&A with @Helge and @Bengo + - assumptions and supersets + - Helga CTA: people are welcome to PR in more docker images to my harness! i need to focus on other work rest of the year + - i'd love it if someone got friendica in there + - bengo: we COULD start connecting my AP work to helge's and stevebate's drivers (by behavior) + - Angelo: web-based "are we _ yet?" site worth making? is that a goal? + - bengo (in chat) + - juan: I would say it's worth doing for the easy tests but realistically couldn't be complete/exhaustive for conformance OR interop purposes