Update HF_TOKEN detection and model download #21

Hugoch · 2025-01-16T22:15:47Z

What does this PR do?

For best security practices HF_TOKEN should be first pulled out from the environment and maybe then from configuration file, this PR ensures it.
Model download is handled manually. This PR relies on huggingface_hub lib to download the model using hf_transfer simplifying the current process.

… configuration.

3outeille · 2025-01-23T14:17:21Z

hi @Hugoch, thanks for your PR, I am going to review it

3outeille · 2025-01-23T16:25:13Z

lgtm !

Hugoch added 10 commits January 16, 2025 16:06

fix: use HF_TOKEN from environment if it exists

b6376a9

fix: use HF_TOKEN from environment if it exists

d053c0f

fix: use HF_TOKEN from environment if it exists

f6d2690

fix: download model even if train.py is called with pre-existing JSON…

4b9d4af

… configuration.

fix: download model even if train.py is called with pre-existing JSON…

223aae4

… configuration.

fix: download model even if train.py is called with pre-existing JSON…

8ea3436

… configuration.

fix: download model even if train.py is called with pre-existing JSON…

93eec72

… configuration.

fix: download model even if train.py is called with pre-existing JSON…

b02fabf

… configuration.

fix: download model even if train.py is called with pre-existing JSON…

d88626d

… configuration.

fix: download model even if train.py is called with pre-existing JSON…

925ba1c

… configuration.

Hugoch requested review from 3outeille and zzhhjjj January 16, 2025 22:15

fix: download model even if train.py is called with pre-existing JSON…

2fd4b54

… configuration.

3outeille merged commit fe88041 into main Jan 23, 2025