Updates to support custom tokenizer and improved RTL display #166
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Backport review apps fixes from feature/cambridge-greek-lexicon
Allows us to build and deploy to review apps without overwriting the "latest" tag. Heroku deployments will remain isolated within the Heroku registry, so there is no chance of "overwriting" a production or staging deploy.
Conditionally apply latest tag
Add hookset and support for overriding tokenizer via get_prepared_tokens
# Conflicts: # backend/requirements.txt
|
@jchill-git I've done this as a PR so you can verify the changes before we merge into your feature branch. If things are working here, please go ahead and approve and merge this branch. If things aren't working, please let me know...happy to try and troubleshoot with you! |
jacobwegner
commented
Jul 19, 2023
| # NOTE: Sample override for The life of Omar ben Saeed below | ||
| if version_urn == "urn:cts:arabicLit:amedsaid1831.dw042.perseus-ara1:": | ||
| from scaife_viewer.atlas.tokenizers import get_tokens_from_csv | ||
| csv_path = Path("data/token-overrides/amedsaid1831.dw042.perseus-ara1.csv") |
There was a problem hiding this comment.
@jchill-git This my "draft" placement for these files; I think we will likely establish a more formal convention in the future.
You can replace my stub file with your actual tokens once you've got them.
jacobwegner
commented
Jul 19, 2023
| from scaife_viewer.atlas.tokenizers import get_tokens_from_csv | ||
| csv_path = Path("data/token-overrides/amedsaid1831.dw042.perseus-ara1.csv") | ||
| return get_tokens_from_csv(version_urn, csv_path) | ||
| return super().get_prepared_tokens(version_urn) |
There was a problem hiding this comment.
(This line will be the fallback, which allows us to use the "default" tokenizer for other texts)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR lands a few things back to the
feature/ibnsaidbranch:Verified locally and in Codespaces:
/reader/urn:cts:arabicLit:amedsaid1831.dw042.perseus-ara1:1?mode=interlinearTo review these changes, you'll want to check out the
ibnsaid-tokenizerbranch in Codespaces and then: