feat: add file lock for remote files download to local path when multiple thread environment. #1887
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…butedType, add data parallel of native mindspore to mindnlp.Trainer.base
Openmind to master
…butedType, add data parallel of native mindspore to mindnlp.Trainer.base
feat: add imdb distributed demo for mindnlp Trainer API
… accelerate package
fix: fix wrong call time for gradient AllReduce
…default PYNATIVE_MODe
lvyufeng
reviewed
Dec 24, 2024
| @@ -20,6 +20,7 @@ echo "start training" | |||
|
|
|||
| export MULTI_NPU="true" | |||
| export ASCEND_SLOG_PRINT_TO_STDOUT=1 | |||
| export ASCEND_RT_VISIBLE_DEVICES=6,7 # Specify the available card numbers | |||
lvyufeng
reviewed
Dec 24, 2024
mindnlp/utils/download.py
Outdated
| lock_file_path = pointer_path + ".lock" | ||
| if sys.platform != "win32": | ||
| # pylint: disable=import-error | ||
| import fcntl |
lvyufeng
reviewed
Dec 24, 2024
mindnlp/utils/download.py
Outdated
| # pylint: disable=import-error | ||
| import fcntl | ||
| else: | ||
| # pylint: disable=import-error |
Closed
…iple thread environment.
…which using package under windows
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
清空.mindnlp执行并行案例可以下载【数据集，tokenizer和pretrainedModel】，0在忙着下载前两个的话，1抢占到文件锁会下载最后一个，都能等待。
thon串行下载也没有问题
对于文件锁，linux环境下无法预编译
winfcntlock.py为winfcntlock.pyc导致pylint会告警E401：impor-error，如果希望兼容windows，pylint告警E0401: Unable to import 是否应该被忽略