[Liger-kernel] Add an option to use _apply_liger_kernel_to_instance() to load model

[hongpeng-guo]

Summary

This PR enables to use Liger Kernel's _apply_liger_kernel_to_instance to init a fsdp worker model.

Main Changes

1. Adding an option of using liger_kernel.transformers.AutoLigerKernelForCausalLM to load a model from pretained, instead of the default transformers.AutoModelForCausalLM
2. Added a test case using configuration file tests/e2e/run_qwen_gsm8k_model_rm_liger_kernel.sh

Related Issue

#96

TODO

#97 optimize the memory usage when computing entropy & log_probs

verl/verl/workers/actor/dp_actor.py

Lines 94 to 106 in 6d96fda

	output = self.actor_module(input_ids=input_ids_rmpad,
	attention_mask=None,
	position_ids=position_ids_rmpad,
	use_cache=False) # prevent model thinks we are generating
	logits_rmpad = output.logits.squeeze(0) # (total_nnz, vocab_size)
	logits_rmpad.div_(temperature)
	# compute entropy
	entropy_rmpad = self.compute_entropy_from_logits(logits_rmpad) # ((total_nnz / sp) + pad)
	# if use_sp: ((total_nnz / sp) + pad) ; if not use_sp: (batch, seqlen)
	log_probs = logprobs_from_logits(logits=logits_rmpad, labels=input_ids_rmpad_rolled)

[corbt]

Personally I would prefer for the job to fail outright if liger is requested but not installed. Just printing a warning is too easy to miss in all the job outputs.

[hongpeng-guo]

I like your proposal. I think we can remove the try ... catch ... logic and add liger_kernel a required dependency of verl. There are still a few extra PRs in Liger kernel targeting for the full integration. Once those are done, I think it makes sense to add liger-kernel to the requirements.txt cc @vermouth1992 @eric-haibin-lin

[vermouth1992]

I guess it's a good idea to make liger-kernel a required dependency and remove try except.

[vermouth1992]

Could you remove try except and run the CI again?

[hongpeng-guo]

Sure, done.

[vermouth1992]

Could you add this test to CI and make sure liger-kernel is toggled?

[hongpeng-guo]

Sure, added this test to .github/workflows/e2e_gsm8k.yml

[vermouth1992]

I guess liger-kernel has to be added to setup.py and project.toml

[hongpeng-guo]

The flag for use_liger is here.

[hongpeng-guo]

A relevant PR for the fused kernel of ce_loss and entropy as described in TODO can be referenced here: linkedin/Liger-Kernel#551

[vermouth1992]

Please run the formatter bash script/format.sh

[hongpeng-guo]

Please run the formatter bash script/format.sh

Got it! Will handle the comments soon. One suggestion is to have a Contributor's Guid section on the readme, or in the doc, i.e., https://docs.sglang.ai/references/contribution_guide.html The contributors would know the checks to run before pushing a PR. Setting up some pre-commit hooks will be better :)

[vermouth1992]

Please run the formatter bash script/format.sh

Got it! Will handle the comments soon. One suggestion is to have a Contributor's Guid section on the readme, or in the doc, i.e., https://docs.sglang.ai/references/contribution_guide.html The contributors would know the checks to run before pushing a PR. Setting up some pre-commit hooks will be better :)

Added a code formatting instruction in the README. Will add more later