2306.08877 Linguistic Binding in Diffusion Models Enhancing Attribute Correspondence through Attention Map Alignment

Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment §

Authors: Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, Gal Chechik

Abstract: Text-conditioned image generation models often generate incorrect associations between entities and their visual attributes. This reflects an impaired mapping between linguistic binding of entities and modifiers in the prompt and visual binding of the corresponding elements in the generated image. As one notable example, a query like “a pink sunflower and a yellow flamingo” may incorrectly produce an image of a yellow sunflower and a pink flamingo. To remedy this issue, we propose SynGen, an approach which first syntactically analyses the prompt to identify entities and their modifiers, and then uses a novel loss function that encourages the cross-attention maps to agree with the linguistic binding reflected by the syntax. Specifically, we encourage large overlap between attention maps of entities and their modifiers, and small overlap with other entities and modifier words. The loss is optimized during inference, without retraining or fine-tuning the model. Human evaluation on three datasets, including one new and challenging set, demonstrate significant improvements of SynGen compared with current state of the art methods. This work highlights how making use of sentence structure during inference can efficiently and substantially improve the faithfulness of text-to-image generation.

What, Why and How §

Here is a summary of the key points from this paper:

What:

• The paper proposes a new method called Syntax-Guided Generation (SyGen) to improve text-to-image generation models.

• SyGen aims to address a common failure mode where attributes specified for one object in the text prompt end up incorrectly applied to a different object in the generated image (improper binding).

Why:

• Current models often fail to properly map linguistic bindings in the text prompt to visual bindings in the generated image.

• The syntactic structure of the prompt is not properly utilized during image generation.

How:

• SyGen first parses the prompt to extract nouns (entities) and their associated modifiers.

• It then defines a novel loss function over the cross-attention maps to encourage high overlap between maps of modifiers and their corresponding nouns, while minimizing overlap between unrelated words.

• This loss is optimized during inference to guide the image generation process based on the syntactic bindings.

• Experiments on three datasets show SyGen significantly improves faithfulness and reduces improper binding compared to previous state-of-the-art methods.

In summary, SyGen leverages syntactic parsing and cross-attention optimization during inference to improve faithfulness of text-to-image generation by properly binding modifiers to entity nouns based on the prompt’s syntactic structure.

Main Contributions §

Here are the main contributions of this paper:

• Proposes Syntax-Guided Generation (SyGen), a new method that incorporates syntactic structure into the text-to-image generation process through optimization of cross-attention maps.

• Defines a novel loss function over the cross-attention maps that encourages alignment between modifiers and their associated nouns based on the parsed syntactic bindings.

• Introduces inference-time optimization of the latent image representation based on this syntactic loss to improve faithfulness.

• Designed a new challenging dataset (DVMP) containing complex coordination sentences with diverse modifiers to evaluate text-to-image generation.

• Demonstrates through human evaluation and analysis that SyGen substantially improves faithfulness and reduces improper binding compared to previous state-of-the-art methods on three datasets.

• Highlights the importance of incorporating syntactic structure into the text-to-image generation process for improving faithfulness.

• Suggests a promising direction for future research by intervening in latent representations using linguistic knowledge like syntactic structure.

In summary, the key contributions are proposing SyGen, a new syntax-guided generation method, a novel syntactic loss function, inference-time optimization, a new challenging dataset, and demonstrating improvements over previous methods by incorporating syntax.

Method Section §

Here is a summary of the method section from the paper:

The method consists of two key components:

1. Identifying modifier-noun relationships:

• The prompt is parsed using a syntactic dependency parser to extract nouns (entities) and their modifiers.

• Modifiers include adjectives, compounds, etc. that are syntactically linked to the noun.

• This results in a set of noun-modifier pairs that reflect the syntactic bindings in the prompt.

2. Controlling generation using cross-attention losses:

• A loss function is defined over the cross-attention maps to encourage alignment between maps of modifiers and corresponding nouns.

• It has two terms: a positive term that maximizes overlap between a noun and its modifiers, and a negative term that minimizes overlap between unrelated words.

• This syntactic loss is used to optimize the latent image during inference by taking gradient steps on the loss.

• The latent is updated for 25 steps out of 50 diffusion steps.

• This guides the diffusion process based on the syntactic structure and encourages proper binding of modifiers to nouns.

In summary, the method parses prompts to extract noun-modifier relationships, defines a novel syntactic loss over cross-attention maps, and intervenes during inference to optimize latents based on this loss to improve visual-linguistic binding.

High-Level Pseudo Code §

Here is the high-level pseudocode for the key components of the method proposed in this paper:

# Parse prompt to get noun-modifier pairs
prompt = "a blue bird and a red fish" 
pairs = syntactic_parser(prompt) 
 
# Pairs would contain:
# [("blue", "bird"), ("red", "fish")]
 
# Diffusion model inference
for t in range(num_diffusion_steps):
  
  # Default denoising step
  latent = denoise(latent) 
  
  if t < num_intervention_steps:
 
    # Get cross attention maps
    attention_maps = get_attention_maps(latent, prompt)
    
    # Define syntactic loss
    loss = syntactic_loss(attention_maps, pairs) 
    
    # Update latent based on loss 
    latent = latent - lr * grad(loss, latent)
 
  # Additional denoising
  latent = denoise(latent) 
 
# Generate image
image = decode(latent)
 
# Syntactic loss function
def syntactic_loss(attention_maps, pairs):
  
  # Positive component
  pos_loss = 0
  for modifier, noun in pairs:
    pos_loss += distance(attention_maps[modifier], attention_maps[noun])
  
  # Negative component 
  neg_loss = 0
  for pair in pairs:
    modifier, noun = pair
    for unrelated in other_words(prompt, pair):
      neg_loss += distance(attention_maps[modifier], attention_maps[unrelated])
      neg_loss += distance(attention_maps[noun], attention_maps[unrelated])
 
  # Final loss
  return pos_loss + neg_loss
 

The key steps are parsing the prompt, defining a syntactic loss over attention maps, intervening to optimize the latent code based on this loss during inference, and generating the image. The loss aligns attention maps between syntactically related words while separating unrelated words.

Detailed Pseudo Code §

Here is a more detailed pseudocode for implementing the key components of Syntax-Guided Generation (SyGen) from the paper:

import Parser, DiffusionModel, Attention
 
# Parse prompt to extract nouns and modifiers
prompt = "a purple cat and a green shoe"
dependencies = Parser.parse(prompt)
nouns, modifiers = extract_entities(dependencies) 
 
# Diffusion model inference
model = DiffusionModel() 
latent = model.init_latent()
for t in range(num_steps):
 
  # Default denoising
  latent = model.denoise(latent)  
 
  # Intervene with syntactic loss
  if t < intervene_until:
    
    # Get attention maps 
    attn_maps = Attention.get_maps(model, latent, prompt)
 
    # Define positive loss 
    pos_loss = 0
    for noun in nouns:
      mods = get_modifiers(noun, modifiers)
      for mod in mods:
        pos_loss += attn_dist(attn_maps[noun], attn_maps[mod])
 
    # Define negative loss
    neg_loss = 0
    for noun in nouns:
      mods = get_modifiers(noun, modifiers)
      others = prompt_words - noun - mods
      for mod in mods:
        for word in others:
          neg_loss += attn_dist(attn_maps[mod], attn_maps[word])
          neg_loss += attn_dist(attn_maps[noun], attn_maps[word])
    
    # Combine losses        
    loss = pos_loss + neg_loss
 
    # Update latent
    latent = latent - lr * latent.grad(loss) 
 
  # Additional denoising
  latent = model.denoise(latent)
 
# Generate image  
img = model.generate(latent)
 
# Attention distance
def attn_dist(a1, a2):
  return sym_kl_div(a1, a2) 
 

The key steps are:

1. Parse prompt to get nouns and modifier words
2. Perform default diffusion denoising
3. Compute attention maps
4. Define positive and negative syntactic loss
5. Update latent based on loss gradient
6. Additional default denoising
7. Generate image

This shows how syntactic structure is incorporated through an attention-optimization based loss to improve visual-linguistic binding.