Are training the CLIP model from scratch? or are you using the pretrained weights?

[Maram-Helmy]

I'm really confuse, in the code you initialize the layers using the normal distribution, but what I understood from the paper is that you are using CLIP model.

Your answer will really help me understand.

Thanks

[hoahoa1808]

As I read code, the CORE_MODEl can be splited into 2 parts:

• the backbone: download and load pretrained CLIP model from OpenAI (read file model/build.py line 17/18)
• the heads: MLM head+cross_modal_transformer and classification head are initialized by the norm dist and trained from scratch.