Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Torchbench timm_regnet training BF16 accuracy regression #1334

Open
mengfei25 opened this issue Feb 5, 2025 · 0 comments
Open

Torchbench timm_regnet training BF16 accuracy regression #1334

mengfei25 opened this issue Feb 5, 2025 · 0 comments
Labels
Accuracy component: torchbench dtype: bfloat16 E2E hw: PVC os: Ubuntu training
Milestone
PT2.8

Comments

@mengfei25
Copy link
Contributor

mengfei25 commented Feb 5, 2025 •
edited
Loading

🐛 Describe the bug

timm_regnet BF16 gor fail accuracy recently, the last known good is b6786e3 + torch a7c2d85

python benchmarks/dynamo/torchbench.py --accuracy --bfloat16 -d xpu -n10 --training --only timm_regnet --backend=inductor

xpu train timm_regnet
E0204 15:16:48.599000 2195001 site-packages/torch/_dynamo/utils.py:2751] RMSE (res-fp64): 0.01081, (ref-fp64): 0.00091 and shape=torch.Size([]). res.dtype: torch.bfloat16, multiplier: 3.000000, tol: 0.001000, use_larger_multiplier_for_smaller_tensor: 0
fail_accuracy

Versions

torch-xpu-ops: b6786e3
python: 3.10
TRITON_COMMIT_ID: e98b6fcb8df5b44eb0d0addb6767c573d37ba024
TORCH_COMMIT_ID: 106acf0eec837d93f7373d894c556bec6cb3c265
TORCHBENCH_COMMIT_ID: 373ffb19dc470f4423a3176a4133f8f4b3cdb5bd
TORCHVISION_COMMIT_ID: d23a6e1664d20707c11781299611436e1f0c104f
TORCHAUDIO_COMMIT_ID: 2709b65c9d3c55da40a5436ec4c45c427feb1d2a
DRIVER_VERSION: 803.61
KERNEL_VERSION: 5.15.0-73-generic #80-Ubuntu SMP Mon May 15 15:18:26 UTC 2023
BUNDLE_VERSION: 2025.0.1.20241113
OS_PRETTY_NAME: Ubuntu 22.04.2 LTS
GCC_VERSION: 11
@mengfei25 mengfei25 mentioned this issue Feb 12, 2025
Merged
github-merge-queue bot pushed a commit that referenced this issue Feb 12, 2025
@mengfei25
Modify CI test (#1358)
c5ac3ec 
1. Update CSV files and skip CSV format check in lintrunner
2. Update lintrunner check for src/comm/DeviceProperties.h
3. Update timm_regnet BF16 training to known fail_accuracy
#1334
@daisyden daisyden added this to the PT2.8 milestone Feb 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Accuracy component: torchbench dtype: bfloat16 E2E hw: PVC os: Ubuntu training
Projects
None yet
Milestone
PT2.8
Development

No branches or pull requests
3 participants
@Stonepia @daisyden @mengfei25