Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Why use zero_module? #64

Open
ZhangMingKun1 opened this issue Sep 17, 2023 · 2 comments
Open

Why use zero_module? #64

ZhangMingKun1 opened this issue Sep 17, 2023 · 2 comments

Comments

@ZhangMingKun1
Copy link

Thanks for your code for the project! It is a really nice work!

I am confused about why using zero_module, may lead to the zero_grad between the input and the output. It is possible to correctly train the model parameter with the expected grad?
@phizaz
Copy link
Owner

phizaz commented Sep 28, 2023

Is it true that zero module is the cause of zero grad? I'm not sure about this.

By the way, we used zero grad module based on a previous work, but by itself, it also has a positive impact faster learning as well (as shown in the previous works).

@ZhangMingKun1
Copy link
Author

Thank you for your feedback! Could you please provide the paper title or GitHub link of the "previous works"?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@phizaz @ZhangMingKun1