Diffusion as an option?

[sethburnette]

How difficult would it be to create a version that uses diffusion like this example?

https://twitter.com/Somnai_dreams/status/1479828948825427968

https://colab.research.google.com/drive/1DHvr-DSNtlpvf0HgI0QmCLSe_PtShzPH?usp=sharing

[rkhamilton]

It’s definitely interesting, but when I started my project everything indicated that clip-guided diffusion had much higher VRAM requirements than VQGAN. From my reading at the time, a GPU with 10GB of VRAM (what I have here) could only render 256x256 pixel images. Since I couldn’t run it effectively locally I don’t think I could do it justice developing a diffusion approach.

[luisarandas]

Hello @rkhamilton!

Any updates on this?

Cheers!

[rkhamilton]

My comment above stands. I don’t like developing in Jupyter notebooks, so if I can’t run it locally I’m not interested in it for a hobby project. My project here is essentially complete as a VGQAN-based library for local art projects.

Philosophically, AI-generated art is a rapidly evolving field, and I think we will keep seeing impressive new algorithms that have different architectures. I knew when I wrote this library that it would soon be left behind by other approaches, and I’m ok with that. Maybe I’ll make a new library later based on a different approach, but it’s not on my to-do list right now.

[luisarandas]

Definitely. Just wondering if you managed to get around the disco diffusion stuff, for now I haven't seen that running on a 8gbVRAM 3070rtx, a bunch of memory allocated to torch and there is more than one model in v5. Still, if you have plans for the future of this library feel free to share. I'm still using it!