MagicMix: Semantic Mixing with Diffusion Models

Jun Hao Liew*
Hanshu Yan*
Daquan Zhou
Jiashi Feng


Teaser figure.

Our MagicMix allows mixing of two different semantics to create a novel concept.


Have you ever imagined what a watermelon-alike lamp or a tiger-alike rabbit would look like? In this work, we attempt to answer these questions by exploring a new task called semantic mixing, aiming at mixing two different semantics to create a new concept (e.g., raccoon and coffee machine). Unlike style transfer where an image is stylized according to the reference style without changing the image content, semantic mixing blends two different concepts in a semantic manner to synthesize a novel concept while preserving the spatial layout and geometry. To this end, we present MagicMix, a simple yet effective solution based on pre-trained text-conditioned diffusion models. Motivated by the progressive property of diffusion models where layout/ shape emerges at early denoising steps while semantically meaningful details appear at later steps during the denoising process, our method first obtains a coarse layout (either by corrupting an image or denoising from a pure Gaussian noise given a text prompt), followed by switching the conditional prompt to the second concept for semantic mixing. Our method does not require any spatial mask or re-training, yet is able to synthesize novel objects (e.g., a watermelon-alike lamp) with high fidelity. With our method, we present our results over diverse downstream applications, including semantic style transfer, novel object synthesis, breed mixing, and concept removal, demonstrating the flexibility of our method.

Semantic Mixing

Semantic mixing refers to the task of mixing two different semantics to create a new concept.

(Hover mouse to reveal the generated results).

+ coffee machine =
corgi + coffee machine
+ tiger =
rabbit + tiger
two way sign
+ propose =
two way sign + propose

Mini Game

We challenge the viewer to guess the involved concepts behind the generated image. Hover mouse to reveal the answer

pagoda + lamp

+ lamp

avocado + fan

+ electric fan

owl + coffee machine

+ coffee machine

pineapple + grapes

+ grapes

rabbit + bag

+ bag

cake + purse

+ purse

phone + vinyl

+ vinyl

rabbit + tiger

+ cat

Concept removal

Image concept can be removed to generate an image with similar layout but with completely different content.

(Hover mouse to check out the concept removed outputs)


remove dumpling

remove dumpling

remove shell

remove shell

remove dog

remove corgi

remove hamburger

remove hamburger

remove watermleon

remove watermelon

remove cat

remove cat

remove pumpkin

remove pumpkin

remove basket

remove fruit basket

More Results

Semantic style transfer

Semantic style transfer results

Novel object synthesis

Novel object synthesis results

Breed mixing

Breed mixing results

Concept removal

Concept removal results


Paper thumbnail

MagicMix: Semantic Mixing with Diffusion Models

Jun Hao Liew*, Hanshu Yan*, Daquan Zhou and Jiashi Feng

arXiv, 2022. (*equal contribution)

    title = {MagicMix: Semantic Mixing with Diffusion Models},
    author = {Liew, Jun Hao and Yan, Hanshu and Zhou, Daquan and Feng, Jiashi},
    journal = {arXiv preprint arXiv:2210.16056},
    year = {2022},


This template was originally made by Phillip Isola and Richard Zhang for a colorful project, and inherits the modifications made by Jason Zhang and Elliott Wu.