Questions about projection layer

[Wgkai]

https://github.com/rese1f/MovieChat/blob/a2d0da02f6cf16383ae1f9891ecf62e6f8f798a5/MovieChat/models/moviechat.py#L156C9-L176C1

        logging.info('Loading LLAMA proj')
        self.llama_proj = nn.Linear(
            self.Qformer.config.hidden_size, self.llama_model.config.hidden_size
        )
        if llama_proj_model:
            print("load llama proj weight: {}".format(llama_proj_model))
            llama_proj_weight = torch.load(llama_proj_model, map_location="cpu")
            msg = model.load_state_dict(llama_proj_weight['model'], strict=False)


        if frozen_llama_proj:
            #  todo frozen  llama_proj
            for name, param in self.llama_proj.named_parameters():
                param.requires_grad = False
            logging.info('LLAMA proj is frozen')
        else:
            for name, param in self.llama_proj.named_parameters():
                param.requires_grad = True
            logging.info('LLAMA proj is not frozen')


        logging.info('Loading llama_proj Done')

I notice that although I specified llama_proj_model in the config file, the parameter llama_proj_model passed in is still ' '.That is to say self.llama_proj is randomly initialized and the weights are not loaded successfully?

[Espere-1119-Song]

Sorry for the confusing code. In fact, the weight of projection layer is loaded after the model initialized.

[xuzq23]

I encounter a similar confusion,
the model path setting in the eval_configs/MovieChat.yaml

  llama_proj_model: 'ckpt/minigpt4/pretrained_minigpt4.pth'
  ckpt: "ckpt/finetune-vicuna7b-v2.pth"

according to your explanation above, would the llama_proj_model and ckpt(line 690) take the same effect?

[Espere-1119-Song]

Yes, they take the same effect.llama_proj_model seems not used in the code, and we will revise it.

[xuzq23]

Yes, they take the same effect.llama_proj_model seems not used in the code, and we will revise it.

Thank you sincerely.