-
Notifications
You must be signed in to change notification settings - Fork 65
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Offset position "longer" than text #83
Comments
Your problem is indeed reproducible with the following code: from language_tool_python import LanguageTool
language = "ES"
tool = LanguageTool(language)
text = 'Rider > Lost Items > Standard lost item > Driver found riders itemdescripcion del articulo perdido 𝑴𝒆 𝒐𝒍𝒗𝒊𝒅𝒆 𝒖𝒏𝒂 𝒎𝒐𝒄𝒋𝒊𝒍𝒂 𝒏𝒆𝒈𝒓𝒔 ingresa un numero de telefono alternativo incluye el codigo de tu pais informacion sobre el viaje 𝒂𝒏𝒅𝒓𝒆𝒔𝒊𝒕𝒐 𝒚 𝒛𝒂𝒑𝒊𝒐𝒍𝒂 podemos compartir tus telefonos con el conductor 𝑺𝒊'
print("len(text)", len(text))
matches = tool.check(text)
for match in matches:
print(match)
corrected_text = language_tool_python.utils.correct(text, matches)
print(corrected_text)
tool.close() Note that some words of your text seem to have special formatting. text_cleaned = 'Rider > Lost Items > Standard lost item > Driver found riders itemdescripcion del articulo perdido Me olvide una mocjila negrs ingresa un numero de telefono alternativo incluye el codigo de tu pais informacion sobre el viaje andresito y zapiola podemos compartir tus telefonos con el conductor Si' |
Seems resolved in #94. Updating shortly. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I have a match that look like this:
Match({'ruleId': 'MORFOLOGIK_RULE_ES', 'message': 'Se ha encontrado un posible error ortográfico.', 'replacements': ['teléfonos', 'teléfono', 'telefotos'], 'offsetInContext': 43, 'context': '...𝒚 𝒛𝒂𝒑𝒊𝒐𝒍𝒂 podemos compartir tus telefonos con el conductor 𝑺𝒊', 'offset': 307, 'errorLength': 9, 'category': 'TYPOS', 'ruleIssueType': 'misspelling', 'sentence': 'Rider > Lost Items > Standard lost item > Driver found riders itemdescripcion del articulo perdido 𝑴𝒆 𝒐𝒍𝒗𝒊𝒅𝒆 𝒖𝒏𝒂 𝒎𝒐𝒄𝒋𝒊𝒍𝒂 𝒏𝒆𝒈𝒓𝒔 ingresa un numero de telefono alternativo incluye el codigo de tu pais informacion sobre el viaje 𝒂𝒏𝒅𝒓𝒆𝒔𝒊𝒕𝒐 𝒚 𝒛𝒂𝒑𝒊𝒐𝒍𝒂 podemos compartir tus telefonos con el conductor 𝑺𝒊'})
Original sentence look like shown in example:
Rider > Lost Items > Standard lost item > Driver found riders itemdescripcion del articulo perdido 𝑴𝒆 𝒐𝒍𝒗𝒊𝒅𝒆 𝒖𝒏𝒂 𝒎𝒐𝒄𝒋𝒊𝒍𝒂 𝒏𝒆𝒈𝒓𝒔 ingresa un numero de telefono alternativo incluye el codigo de tu pais informacion sobre el viaje 𝒂𝒏𝒅𝒓𝒆𝒔𝒊𝒕𝒐 𝒚 𝒛𝒂𝒑𝒊𝒐𝒍𝒂 podemos compartir tus telefonos con el conductor 𝑺𝒊
Problem is that offset is said to be 307, while sentence length in chars 296.
I think that the problem is that the text has some chars that actually internally take more than one position in unicode encoding (are compose but multiple chars).
The problem is that when I try to reference detection to original text I get an error because that position is wrong and does not reference the true position in the text
The text was updated successfully, but these errors were encountered: