[Issue]: Corrupted Output in CogVideoX image to video

[holgerpieta]

Issue Description

After temporarily fixing #3627 video generation finishes, but the result is corrupted, as discussed here: #3625 (comment)

Nothing unexpected can be seen in the debug logs, no warning or error messages.

Version Platform Description

15:43:41-548170 INFO     Starting SD.Next
15:43:41-550171 INFO     Logger: file="D:\SD.Next\automatic\sdnext.log" level=DEBUG size=65 mode=create
15:43:41-552172 INFO     Python: version=3.11.9 platform=Windows bin="D:\SD.Next\automatic\venv\Scripts\python.exe"
                         venv="D:\SD.Next\automatic\venv"
15:43:41-744339 INFO     Version: app=sd.next updated=2024-11-22 hash=6846f4e5 branch=master
                         url=https://github.com/vladmandic/automatic/tree/master ui=main
15:43:42-433365 INFO     Repository latest available a2f5a39a5919d7edae9b4825a6551c50c854c504 2024-12-09T08:27:14Z
15:43:42-439369 INFO     Platform: arch=AMD64 cpu=AMD64 Family 25 Model 33 Stepping 2, AuthenticAMD system=Windows
                         release=Windows-10-10.0.22631-SP0 python=3.11.9 docker=False
15:43:42-441369 DEBUG    Packages: venv=venv site=['venv', 'venv\\Lib\\site-packages']
15:43:42-442369 INFO     Args: ['--use-zluda', '--debug', '--autolaunch', '--docs']
15:43:42-442369 DEBUG    Setting environment tuning
15:43:42-443370 DEBUG    Torch allocator: "garbage_collection_threshold:0.80,max_split_size_mb:512"
15:43:42-447370 DEBUG    Torch overrides: cuda=False rocm=False ipex=False directml=False openvino=False zluda=True
15:43:42-457372 INFO     Python: version=3.11.9 platform=Windows bin="D:\SD.Next\automatic\venv\Scripts\python.exe"
                         venv="D:\SD.Next\automatic\venv"
15:43:42-461374 INFO     ROCm: AMD toolkit detected
15:43:42-802205 INFO     ROCm: agents=['gfx1102']
15:43:42-803205 INFO     ROCm: version=5.7, using agent gfx1102
15:43:42-836717 INFO     Using ZLUDA in D:\SD.Next\automatic\.zluda
No ROCm runtime is found, using ROCM_HOME='C:\Program Files\AMD\ROCm\5.7'
15:43:44-219274 WARNING  Modified files: ['models/Reference/black-forest-labs--FLUX.1-dev.jpg', 'scripts/cogvideo.py']
15:43:44-264787 INFO     Install: verifying requirements
15:43:44-267788 INFO     Verifying packages
15:43:44-311799 DEBUG    Timestamp repository update time: Fri Nov 22 19:01:15 2024
15:43:44-312800 INFO     Startup: standard
15:43:44-312800 INFO     Verifying submodules
15:43:46-278547 DEBUG    Git submodule: extensions-builtin/sd-extension-chainner / main
15:43:46-401079 DEBUG    Git detached head detected: folder="extensions-builtin/sd-extension-system-info" reattach=main
15:43:46-403080 DEBUG    Git submodule: extensions-builtin/sd-extension-system-info / main
15:43:46-476475 DEBUG    Git submodule: extensions-builtin/sd-webui-agent-scheduler / main
15:43:46-589051 DEBUG    Git detached head detected: folder="extensions-builtin/sdnext-modernui" reattach=main
15:43:46-590050 DEBUG    Git submodule: extensions-builtin/sdnext-modernui / main
15:43:46-696579 DEBUG    Git submodule: extensions-builtin/stable-diffusion-webui-rembg / master
15:43:46-775100 DEBUG    Git submodule: modules/k-diffusion / master
15:43:46-885631 DEBUG    Git detached head detected: folder="wiki" reattach=master
15:43:46-886631 DEBUG    Git submodule: wiki / master
15:43:46-955150 DEBUG    Register paths
15:43:46-999161 DEBUG    Installed packages: 186
15:43:47-000161 DEBUG    Extensions all: ['Lora', 'sd-extension-chainner', 'sd-extension-system-info',
                         'sd-webui-agent-scheduler', 'sdnext-modernui', 'stable-diffusion-webui-rembg']
15:43:47-124695 DEBUG    Extension installer:
                         D:\SD.Next\automatic\extensions-builtin\sd-webui-agent-scheduler\install.py
15:43:49-068524 DEBUG    Extension installer:
                         D:\SD.Next\automatic\extensions-builtin\stable-diffusion-webui-rembg\install.py
15:43:56-466680 INFO     Extension installed packages: stable-diffusion-webui-rembg ['rembg==2.0.61']
15:43:56-467681 DEBUG    Extensions all: []
15:43:56-468682 INFO     Extensions enabled: ['Lora', 'sd-extension-chainner', 'sd-extension-system-info',
                         'sd-webui-agent-scheduler', 'sdnext-modernui', 'stable-diffusion-webui-rembg']

Relevant log output

15:49:58-567233 INFO     CogVideoX load: model="THUDM/CogVideoX-5b-I2V"
Diffusers ?it/s                    0% 0/5 00:00 ? Loading pipeline components...15:49:59-811857 DEBUG    Server: alive=True requests=66 memory=3.28/95.91 status='idle' task='' timestamp=None
                         id='task(02td4u3m26uj15f)' job=0 jobs=0 total=1 step=0 steps=0 queued=0 uptime=360
                         elapsed=355.77 eta=None progress=0
Diffusers  4.04it/s █████████████ 100% 2/2 00:00 00:00 Loading checkpoint shards
Diffusers  1.89it/s ████████ 100% 5/5 00:02 00:00 Loading pipeline components...
15:50:01-774474 DEBUG    CogVideoX load: class="CogVideoXImageToVideoPipeline"
15:50:01-775979 DEBUG    CogVideoX: offload=balanced
15:50:01-798985 INFO     CogVideoX: sampler=DDIM steps=20 frames=5 width=720 height=480 seed=2675881999 guidance=6
15:50:01-800986 DEBUG    CogVideoX input: image=<PIL.Image.Image image mode=RGB size=720x480 at 0x26C2208C350>
15:50:01-801986 INFO     CogVideoX: class=CogVideoXImageToVideoPipeline frames=5 input=<PIL.Image.Image image mode=RGB
                         size=720x480 at 0x26C2208C350>
Progress 82.72s/it █▉                                       5% 1/20 01:22 26:11 15:51:59-559656 DEBUG    Server: alive=True requests=185 memory=8.71/95.91 status='idle' task='' timestamp=None
                         id='task(02td4u3m26uj15f)' job=0 jobs=0 total=1 step=0 steps=0 queued=0 uptime=479
                         elapsed=475.52 eta=None progress=0
Progress 74.57s/it ███▊                                    10% 2/20 02:31 22:22 15:53:29-590609 DEBUG    VAE load: type=approximate model="models\VAE-approx\model.pt"
Progress 71.91s/it █████▋                                  15% 3/20 03:40 20:22 15:54:00-157091 DEBUG    Server: alive=True requests=272 memory=8.65/95.91 status='idle' task='' timestamp=None
                         id='task(02td4u3m26uj15f)' job=0 jobs=0 total=1 step=0 steps=0 queued=0 uptime=600
                         elapsed=596.12 eta=None progress=0
Progress 70.74s/it ███████▌                                20% 4/20 04:49 18:51 15:56:00-119169 DEBUG    Server: alive=True requests=394 memory=8.66/95.91 status='idle' task='' timestamp=None
                         id='task(02td4u3m26uj15f)' job=0 jobs=0 total=1 step=0 steps=0 queued=0 uptime=720
                         elapsed=716.08 eta=None progress=0
Progress 69.68s/it ███████████▍                            30% 6/20 07:07 16:15 15:57:59-885515 DEBUG    Server: alive=True requests=516 memory=8.65/95.91 status='idle' task='' timestamp=None
                         id='task(02td4u3m26uj15f)' job=0 jobs=0 total=1 step=0 steps=0 queued=0 uptime=840
                         elapsed=835.85 eta=None progress=0
Progress 70.06s/it ███████████████▏                        40% 8/20 09:27 14:00 15:59:59-829901 DEBUG    Server: alive=True requests=631 memory=8.6/95.91 status='idle' task='' timestamp=None
                         id='task(02td4u3m26uj15f)' job=0 jobs=0 total=1 step=0 steps=0 queued=0 uptime=960
                         elapsed=955.79 eta=None progress=0
Progress 70.00s/it █████████████████                       45% 9/20 10:37 12:49 16:01:59-710211 DEBUG    Server: alive=True requests=749 memory=8.63/95.91 status='idle' task='' timestamp=None
                         id='task(02td4u3m26uj15f)' job=0 jobs=0 total=1 step=0 steps=0 queued=0 uptime=1079
                         elapsed=1075.67 eta=None progress=0
Progress 69.90s/it ████████████████████▎                  55% 11/20 12:57 10:29 16:04:00-336581 DEBUG    Server: alive=True requests=879 memory=8.67/95.91 status='idle' task='' timestamp=None
                         id='task(02td4u3m26uj15f)' job=0 jobs=0 total=1 step=0 steps=0 queued=0 uptime=1200
                         elapsed=1196.3 eta=None progress=0
Progress 69.21s/it ████████████████████████               65% 13/20 15:14 08:04 16:06:00-434233 DEBUG    Server: alive=True requests=1011 memory=8.61/95.91 status='idle' task='' timestamp=None
                         id='task(02td4u3m26uj15f)' job=0 jobs=0 total=1 step=0 steps=0 queued=0 uptime=1320
                         elapsed=1316.4 eta=None progress=0
Progress 68.39s/it ███████████████████████████▊           75% 15/20 17:29 05:41 16:07:59-821705 DEBUG    Server: alive=True requests=1131 memory=8.83/95.91 status='idle' task='' timestamp=None
                         id='task(02td4u3m26uj15f)' job=0 jobs=0 total=1 step=0 steps=0 queued=0 uptime=1440
                         elapsed=1435.78 eta=None progress=0
Progress 67.69s/it ███████████████████████████████▍       85% 17/20 19:44 03:23 16:10:00-169868 DEBUG    Server: alive=True requests=1251 memory=8.67/95.91 status='idle' task='' timestamp=None
                         id='task(02td4u3m26uj15f)' job=0 jobs=0 total=1 step=0 steps=0 queued=0 uptime=1560
                         elapsed=1556.13 eta=None progress=0
Progress 67.41s/it █████████████████████████████████▎     90% 18/20 20:50 02:14 16:11:59-719887 DEBUG    Server: alive=True requests=1369 memory=8.67/95.91 status='idle' task='' timestamp=None
                         id='task(02td4u3m26uj15f)' job=0 jobs=0 total=1 step=0 steps=0 queued=0 uptime=1679
                         elapsed=1675.68 eta=None progress=0
Progress 69.21s/it █████████████████████████████████████ 100% 20/20 23:04 00:00
16:14:00-004236 DEBUG    Server: alive=True requests=1457 memory=40.92/95.91 status='idle' task='' timestamp=None
                         id='task(02td4u3m26uj15f)' job=0 jobs=0 total=1 step=0 steps=0 queued=0 uptime=1800
                         elapsed=1795.97 eta=None progress=0
16:14:52-405413 INFO     CogVideoX: frame=<PIL.Image.Image image mode=RGB size=720x480 at 0x26C22E5D690> frames=8
                         its=0.11 time=1490.60
16:14:54-187207 DEBUG    GC: utilization={'gpu': 100, 'ram': 42, 'threshold': 80} gc={'collected': 325, 'saved': 0.0}
                         before={'gpu': 15.98, 'ram': 40.35} after={'gpu': 15.98, 'ram': 7.59, 'retries': 0, 'oom': 0}
                         device=cuda fn=run:run time=1.78
16:14:54-423956 DEBUG    GC: utilization={'gpu': 100, 'ram': 8, 'threshold': 80} gc={'collected': 127, 'saved': 0.0}
                         before={'gpu': 15.98, 'ram': 7.59} after={'gpu': 15.98, 'ram': 7.59, 'retries': 0, 'oom': 0}
                         device=cuda fn=run:end time=0.23
16:14:54-641018 DEBUG    GC: utilization={'gpu': 100, 'ram': 8, 'threshold': 80} gc={'collected': 159, 'saved': 0.0}
                         before={'gpu': 15.98, 'ram': 7.59} after={'gpu': 15.98, 'ram': 7.59, 'retries': 0, 'oom': 0}
                         device=cuda fn=f:end time=0.22
16:14:54-770159 DEBUG    Save temp: image="C:\Users\Pieta\AppData\Local\Temp\gradio\tmpuh6up6p3.png" width=720
                         height=480 size=508373
16:14:54-899198 DEBUG    Save temp: image="C:\Users\Pieta\AppData\Local\Temp\gradio\tmp_6pnzl8n.png" width=720
                         height=480 size=507256
16:14:55-030179 DEBUG    Save temp: image="C:\Users\Pieta\AppData\Local\Temp\gradio\tmpmnayi4ls.png" width=720
                         height=480 size=497238
16:14:55-168080 DEBUG    Save temp: image="C:\Users\Pieta\AppData\Local\Temp\gradio\tmpj2vizg1f.png" width=720
                         height=480 size=494760
16:14:55-279129 DEBUG    Save temp: image="C:\Users\Pieta\AppData\Local\Temp\gradio\tmp6qiu8dhk.png" width=720
                         height=480 size=539304
16:14:55-377233 DEBUG    Save temp: image="C:\Users\Pieta\AppData\Local\Temp\gradio\tmput1vw51c.png" width=720
                         height=480 size=575494
16:14:55-469256 DEBUG    Save temp: image="C:\Users\Pieta\AppData\Local\Temp\gradio\tmprb8d14b8.png" width=720
                         height=480 size=595786
16:14:55-552784 DEBUG    Save temp: image="C:\Users\Pieta\AppData\Local\Temp\gradio\tmpso4hnyry.png" width=720
                         height=480 size=602533

Backend

Diffusers

UI

Standard

Branch

Master

Model

Other

Acknowledgements

• I have read the above and searched for existing issues
• I confirm that this is classified correctly and its not an extension issue