You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I try to parse code that include comment with Japanese and Chinese.
when I want to get all tokens spelling in a method by iterate tokens in cursor.get_tokens(), it will raise error when it try to decode some string into UTF-8 in default.
what if I got a token and want to decode it into other format like "shiftJIS"?
can add some args like "encoding" when return token.spelling ?
The text was updated successfully, but these errors were encountered:
I try to parse code that include comment with Japanese and Chinese.
when I want to get all tokens spelling in a method by iterate tokens in cursor.get_tokens(), it will raise error when it try to decode some string into UTF-8 in default.
what if I got a token and want to decode it into other format like "shiftJIS"?
can add some args like "encoding" when return token.spelling ?
The text was updated successfully, but these errors were encountered: