A VFX artist’s perspective on generating AI assisted images with ComfyUI

I’ve put together this longer form video on what I know regarding AI assisted image generation from a VFX artist's perspective and where I think the tech is going.

I’ve also in the last couple days been working on improving this workflow to produce a pipeline that could generate finalized shots. Unfortunately, it's currently not feasible at the moment. Creating stable diffusion models that are compatible with motion vector AOV rendered images is the key requirement to make this a reality. It's highly probable that this could happen within the coming months.
As I’ve said in previous posts I don’t think there is much of a negative impact on a typical VFX pipeline as it largely stays the same. Compositors get a new set of tools, or it’s branched out as a dedicated AI artist department. They would mostly be working with node based tools to comp latent space images instead of pixel space. Many 3d departments in a VFX pipeline stay the same. There is less load on the lighting/rendering department which is always a bottle neck. Texture/shading departments are less focused on reaching realism and can think conceptually what texture and shading makes for an aesthetically correct image that could not be represented as prompts.

As you can see in this video when changing to a sunny day from overcast, it makes plausible shadows but they are technically incorrect. Such things like this would be compounded when moving to a full shot/image sequence.

These ideas of what a future VFX pipeline also align well with the virtual production because stable diffusion generated images can be baked back to 3d using gaussian splatting and rendered realtime in Unreal Engine. This brings a more creative artistic oriented workflow of making photo real 3d assets as another option that is comparable to real objects scanned with photogrammetry.

A lot of artists are nervous about AI tools replacing them or making the entire industry obsolete but I feel the reality now can be summed up as, A picture is worth a thousand latent prompts. How many prompts would it take to describe a 2 hour feature film? If one could be distilled down to just words. It’s been shown from the masses what it can do in the hands of a non-artist with prompts but the results tend to alway look generic.

If any AI/ML developers working with diffusion models want to work with me on this, feel free to reach out. Having been away from VFX for a while and currently unemployed due to the actors and writers strike, I'm fortunate not to be restricted by any NDA to publicly talk about this stuff. This comes at a time when AI is poised to rapidly enter the VFX industry next year as I feel it is a true artist friendly tool now.

🔔 Subscribe for more DIY CNC content:    / @bryanhoward  

Download the source files and workflow here:
https://drive.google.com/file/d/1i-ft...

If you enjoy my DIY CNC projects, consider supporting the channel on Patreon. Your support helps me create more content and improve the channel. Thank you!
🔗 Patreon: https://www.patreon.com/user?u=56767327

0:00 Intro
2:47 Maya scene
3:33 3d modeling
4:36 Single Leg Rigging
5:32 Body Rigging
7:34 Hexapod Gait Animation
11:19 Render Passes
14:26 Compositing Element and AOV prep
19:08 ComfyUI Workflow overview

#genai #vfx #comfyui