Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace Python parser with ruff parser #5453

Closed
wants to merge 44 commits into from
Closed

Replace Python parser with ruff parser #5453

wants to merge 44 commits into from

Conversation

etaloof
Copy link

@etaloof etaloof commented Dec 2, 2024

This PR replaces the Python parser with the ruff Parser. It is based on the work of the earlier PR #5423 (thanks @qingshi163)! Code generation has been updated and should mostly work now. There is is still some work left to fix the built-in AST module, REPL (probably) and I haven't looked into WASM support.

Running the tests with cargo test --workspace --exclude rustpython_wasm --features freeze-stdlib gives me one remaining test failure: test_run_script in src/lib.rs. It seems fail when importing the extra_tests/snippets/dir_main but I am not sure why that happens.

Concerning the AST module: The previous implementation is using a lot of generated code (script). Not sure if we should continue updating this script, I'll look into it. @youknowone do you have any suggestions on this?

qingshi163 and others added 21 commits September 10, 2024 19:09
@qingshi163
stage1
93b7d6b
@qingshi163
compiler pass build
e0a3f26
@qingshi163
introduce rustpython-compiler-source
38575bd
@qingshi163
stage2
dce0d94
@qingshi163
fixup
3162b9b
@qingshi163
pass compile
1f7024a
@etaloof
Fix hello world compiler test
e189ecf
@etaloof
Fix code generation for if-elif-else statement
0f100e9
@etaloof
Fix code generation for lambda expression
b42f88f
@etaloof
Fix code generation for integers
7c53857
@etaloof
Fix code generation for fstrings
424e05a
@etaloof
Fix code generation for if statement
2691cbe
@etaloof
Fix code generation for if statement
a98a623
@etaloof
Fix code generation for if statement
168aa01
@etaloof
Fix code generation for fstring
c25d958
@etaloof
Fix code generation for class definition
4e29dd4
@etaloof
Replace feature flags
89afad2
@etaloof
Initialize frozen core modules
e760ac7
@etaloof
Allow __future__ import after module doc comment
2987cbe
@etaloof
Disable ast module
c10df16
@etaloof
Commit remaining fixes for compile errors in examples
bcfbf88
@youknowone
Copy link
Member

Thank you so much! Since ruff already has its own ast coworking with same parser, how about starting from ruff side again about it too?
I couldnt look in details yet, but i feel like code generated ast module is not cost efficient anymore. We now can borrow lots of code from other repository easy

@youknowone
Copy link
Member

@qingshi163 @coolreader18 do you also have advices?

@youknowone
Copy link
Member

I have to look in this PR, sorry for delay

@youknowone youknowone mentioned this pull request Jan 18, 2025
Open
@etaloof
Update ast stdlib module
1bd4078
@etaloof
Update ast stdlib module
a054ca6
@etaloof
Reset barebones example
1476b49
@etaloof
Merge remote-tracking branch 'refs/remotes/origin/main' into ruff-parser
1a3c5bb 
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	compiler/codegen/src/compile.rs
#	compiler/codegen/src/symboltable.rs
#	wasm/lib/src/convert.rs
@etaloof
Copy link
Author

etaloof commented Jan 25, 2025

@youknowone I think this is ready to be reviewed now. I went ahead and merged main into this branch to resolve the merge conflicts (see commit 1a3c5bb. Not sure if this is the correct way to do things, I can rebase and merge again if needed.

@youknowone
Copy link
Member

@etaloof could you make your fork of RustPython not from qingshi163's but from RustPython org's? Then I can directly push my edits to your PR

@youknowone
Copy link
Member

It looks going well. this commit will pin parser version to git version of Ruff c8acbf8

youknowone
youknowone reviewed Jan 27, 2025
View reviewed changes
Copy link
Member

@youknowone youknowone left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, I feel like everything is going to the right direction.
Is there any blocking issue?

compiler/core/src/mode.rs Show resolved Hide resolved
vm/src/stdlib/ast/basic.rs Show resolved Hide resolved
compiler/codegen/src/compile.rs Show resolved Hide resolved
@etaloof etaloof closed this by deleting the head repository Jan 27, 2025
@etaloof
Copy link
Author

etaloof commented Jan 27, 2025 •
edited
Loading

Thank you, I deleted my fork from qingshi163's repository and forked again from RustPython. Could you please open this PR again? Sorry for the confusion.

@youknowone
Copy link
Member

a PR is only can be re-opened when the branch is not changed. This one doesn't seem so. Please open a new one.

This was referenced Jan 28, 2025
Closed
Open
@etaloof
Copy link
Author

etaloof commented Jan 28, 2025

Thanks, I have created a new PR #5494.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@youknowone youknowone youknowone left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@etaloof @youknowone @qingshi163