Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for APOLLO optimizer #668

Open
ddaspit opened this issue Feb 27, 2025 · 0 comments
Open

Add support for APOLLO optimizer #668

ddaspit opened this issue Feb 27, 2025 · 0 comments
Labels
pipeline 4: train Issue related to training a model.

Comments

@ddaspit
Copy link
Collaborator

ddaspit commented Feb 27, 2025

The APOLLO optimizer has been added to Huggingface Transformers. APOLLO is a memory-efficient optimizer with similar performance to AdamW. We will need to add a dependency on the apollo-torch package. APOLLO can be enabled by setting the optim to apollo_adamw. We will also need to automatically set optim_target_modules based on the currently selected model. For more information, see the documentation.

@ddaspit ddaspit added the pipeline 4: train Issue related to training a model. label Feb 27, 2025
@ddaspit ddaspit moved this from 🆕 New to 🔖 Ready in SIL-NLP Research Feb 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pipeline 4: train Issue related to training a model.
Projects
Status: 🔖 Ready
Development

No branches or pull requests

1 participant