Relevancy Delta Review

Work in Progress.

About

When development or operations work changes the Searchworking Solr index, there may be changes to UX aspects of the service like new relevancy rankings, or changes in results returned for known searches.

To balance the need to keep Solr and index resources up to date & able to take on new features with the need to keep a consistent and reviewed Searchworks search results user experience, use the following "relevancy delta review". This relevancy delta review enables feedback from identified Searchworks super users to be taken into consideration by DLSS teams before deploying changes to production. It is a process to be used at the discretion of the a team lead in consultation with identified key stakeholders.

Please adapt these steps as needed for each work cycles' particular needs. This should be just a starting point for unblocking index resource changes while keeping SW Index consistency.

What is the Relevancy Delta

The following only measures deltas in relevancy across Solr indices (normally, SW Production Index and a proposed dev index), not the quality of relevancy in a given index. Deltas can be good or needed improvements, but still may need review before shipping.

Currently, this only includes MARC data (and that should be noted in the super user group engagement done below). We hope to wrap in MODS data within the ongoing Summer 2018 Searchworks Indexing & Performance work cycle.

Relevancy Delta Review

1. When the changed resources (codebases, configs, solr indices, or other) are ready for review, changes should be deployed to the Searchworks staging & review environments.
   • currently, sw-webapp-sandbox-e.stanford.edu which points to sw-relevancy index on sw-solr, and sw-webapp-sandbox-f.stanford.edu which points to searchworks-dev index on sul-solr.
   • when preparing for hand-off to the Searchworks super users group, create some vanity URLs / aliases for these.
2. If not already done as part of the dev process, run the sw_index_tests suite on the new index to identify any changes to identified issues.
3. If not already done as part of the dev process, run the last month's top 1000 (or other subset of) queries through SW Relevancy Delta Dashboard & identify any divergences that need to be accounted for (they may be improvements & should just be highlighted).
4. Team should identify & recruit the Searchworks super users group appropriate for the project at hand. You probably want to plan at least a few weeks out to enable scheduling as needed for that group of people.
5. The identified Searchworks super users group should be emailed with the following info:
   • Non-technical explanation of what changes led to this review (e.g. "The Solr index has been updated to a new Solr version. This has some effects on things like relevancy and results rankings. We want you to assess the changes."
   • Some known points of change the super users group may watch out for. This should be a bullet point list of simplified explanations of things we know have changed & explain the reasons (if known). The bullet point list should include information from sw-solr-tests and the SW Relevancy Delta Dashboard, but those resources should not be shared directly (i.e. the test suite & dashboard are team internal tools).
   • Amount of time for the review (e.g. "We need this reviewed within the next 5 days in order to act on your feedback within this work cycle.")
   • URLs for resources along with brief descriptions of what they are (using the vanity urls generated in step 1);
   • Communication channels to use for feedback (e.g. searchworks-feedback if you want to involve that larger group; a single point of contact [e.g. the PO or TL or stakeholder on the team] for targeted feedback).
   • Indication that they should also perform their own testing processes & domain-specific searches even if not identified above. These might eventually feed into extending our test suite.
6. After gathering feedback, the project team should review for any possible blockers and plan how to address.
7. There should be communication back to the super users group to indicate how feedback was acted upon.