AI for design workflows

Ar. Albert Sumin shares his experience on how AI has transformed his design process, from something as simple as inserting furniture to changing façade materials to something as complex as concept development. He also shares why ComfyUI is his most preferred application.

I first encountered generative AI for image creation in 2021 while working at Coop Himmelb(l)au on a theatre competition project in Kazan. The office had (and still has) an internal neural network called Deep Himmelb(l)au, trained on photos and renderings of the studio’s projects. Daniel Bolojan was responsible for this system. Our competition team used Deep Himmelb(l)au’s generations to explore early design ideas. At the time, tools like DALL·E or Midjourney weren’t publicly available, and we probably didn’t fully realise how groundbreaking and innovative this approach was. Today, similar workflows are standard practice in many studios around the world.

Two years later, in 2023, I began using AI actively in teaching. Together with Eduard Haiman, we developed and ran several workshops. In one, we focused on transforming Midjourney-generated images into 3D scenes. We also used AI to create textures, displacement, and bump maps.

Currently, in my practice and work for architectural studios, I use several AI tools for working with images and videos: Midjourney, Photoshop with built-in Adobe Firefly, Gemini, and ChatGPT — but my main tool is ComfyUI.

ComfyUI is installed locally on your computer. It’s free and open-source and supports many models for generating images, video, audio, text, 3D models, and more. Below are a few examples of what you can do in design and architecture when working with images in ComfyUI.

Example 1: Inserting furniture into an interior photo

Let’s say you have a photo of an empty room and separate images of furniture you want to place there.

You’ll need a relatively complex workflow (what scripts are called in ComfyUI), often built using Flux Tools. But once that’s set up, you only need to draw masks to show where the sofa, table, and rug should go. This technique is called inpainting, and the mask editor is built directly into ComfyUI.

This method isn’t limited to interiors — you can alter or add façade details, insert specific plants or street furniture when working on landscape designs, and much more.

Example 2: Changing façade materials

This task uses a different technique and models, such as SDXL with ControlNet.

AI can apply not only realistic materials – bricks, metal panels, concrete… – but also more imaginative ones. For example, what if the facade were made of soap bubbles, snow, or jellyfish?

While such ideas might not be practical, they can reveal new aesthetics or formal approaches. For instance, I once generated a façade meant to look like it was made of paper. Obviously, real paper can’t be used structurally, but the design’s plasticity could be reinterpreted in other materials.

Example 3: Concept development

AI is a powerful tool for exploring novel forms based on your sketches or ideas. Starting a new project with AI is a fascinating and creative process. 

If you want to adhere to a certain concept or style, you can train a custom model and integrate it into your workflow. I’ve trained many models myself, and several are uploaded to Civitai, the largest platform for diffusion models. There, you’ll find a collection of models, including mine and others shared by architects and designers. Below is an example using a model trained on Coop Himmelb(l)au projects.

Example 4: AI rendering

This includes a wide variety of workflows, depending on the inputs. You can start with almost anything—a photo of crumpled paper, a physical model, a sketch, a massing model, or a detailed 3D model—and turn it into a refined render. The image below shows how your input evolves into a polished result.

A subtask here is editing an existing render — for example, changing a summer scene into winter or enhancing specific details. This is also fully possible.

These are just a few of the tasks you can handle using ComfyUI. In addition to generating images, 3D models, and videos, you can use it for small editing, such as colour correction, background removal, creating textures and patterns, and more.

ComfyUI isn’t tailored specifically for architects or designers — it’s a universal tool that can be used across many disciplines and for working with all types of visual content. This makes it not only powerful but also uniquely flexible for creative professionals.

For more details, visit: https://www.comfy.org/