Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support decode Token into Multi encoding type ? #60

Open
Vela-zz opened this issue Aug 3, 2023 · 1 comment
Open

Support decode Token into Multi encoding type ? #60

Vela-zz opened this issue Aug 3, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@Vela-zz
Copy link

Vela-zz commented Aug 3, 2023

I try to parse code that include comment with Japanese and Chinese.

when I want to get all tokens spelling in a method by iterate tokens in cursor.get_tokens(), it will raise error when it try to decode some string into UTF-8 in default.

what if I got a token and want to decode it into other format like "shiftJIS"?
can add some args like "encoding" when return token.spelling ?

@sighingnow
Copy link
Owner

I will try submit pull request to upstream LLVM.

@sighingnow sighingnow added the enhancement New feature or request label Aug 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants