2306.07754 Generative Watermarking Against Unauthorized Subject-Driven Image Synthesis

Generative Watermarking Against Unauthorized Subject-Driven Image Synthesis §

Authors: Yihan Ma, Zhengyu Zhao, Xinlei He, Zheng Li, Michael Backes, Yang Zhang

Abstract: Large text-to-image models have shown remarkable performance in synthesizing high-quality images. In particular, the subject-driven model makes it possible to personalize the image synthesis for a specific subject, e.g., a human face or an artistic style, by fine-tuning the generic text-to-image model with a few images from that subject. Nevertheless, misuse of subject-driven image synthesis may violate the authority of subject owners. For example, malicious users may use subject-driven synthesis to mimic specific artistic styles or to create fake facial images without authorization. To protect subject owners against such misuse, recent attempts have commonly relied on adversarial examples to indiscriminately disrupt subject-driven image synthesis. However, this essentially prevents any benign use of subject-driven synthesis based on protected images. In this paper, we take a different angle and aim at protection without sacrificing the utility of protected images for general synthesis purposes. Specifically, we propose GenWatermark, a novel watermark system based on jointly learning a watermark generator and a detector. In particular, to help the watermark survive the subject-driven synthesis, we incorporate the synthesis process in learning GenWatermark by fine-tuning the detector with synthesized images for a specific subject. This operation is shown to largely improve the watermark detection accuracy and also ensure the uniqueness of the watermark for each individual subject. Extensive experiments validate the effectiveness of GenWatermark, especially in practical scenarios with unknown models and text prompts (74% Acc.), as well as partial data watermarking (80% Acc. for 1/4 watermarking). We also demonstrate the robustness of GenWatermark to two potential countermeasures that substantially degrade the synthesis quality.

What, Why and How §

Here is a summary of the key points from this paper:

What: The paper proposes a new generative watermarking approach called \textbf{Generative Watermarking Against Unauthorized Subject-Driven Image Synthesis (\MethodName)}. This is designed to prevent misuse of subject-driven synthesis models that can generate personalized images of specific subjects (e.g. a person’s face or an artist’s style).

Why: Existing protections against misuse of subject-driven synthesis rely on adversarial examples, which disrupt all synthesis. This prevents benign use. \MethodName instead allows benign use but prevents unauthorized synthesis.

How:

• \MethodName has a watermark generator and detector. The generator adds imperceptible watermarks to images. The detector checks for watermarks.
• They are jointly trained on large datasets to improve robustness and invisibility.
• The detector is fine-tuned for each subject using their synthesized images. This improves personalized detection and ensures watermark uniqueness.
• Extensive experiments validate effectiveness even with unknown models/prompts (74% accuracy) and partial watermarking (80% for 1/4 images).
• \MethodName is robust to countermeasures like noise injection and compression that degrade image quality.

In summary, \MethodName introduces a new generative watermarking approach to prevent unauthorized subject-driven synthesis while maintaining utility for benign use. Key aspects are joint optimization, personalized fine-tuning, and demonstrated effectiveness in challenging scenarios.

Main Contributions §

Here are the main contributions of this paper:

1. Proposes Generative Watermarking Against Unauthorized Subject-Driven Image Synthesis (\MethodName), the first watermarking approach for protecting images against misuse in subject-driven synthesis while maintaining utility.

2. Develops a framework with a jointly trained generator and detector. The generator produces robust and invisible watermarks. The detector is fine-tuned for each subject using synthesized images to improve personalized detection.

3. Demonstrates effectiveness of \MethodName in two synthesis tasks (artistic style and human face) using two models (Textual Inversion and DreamBooth). Achieves high accuracy even with unknown models/prompts and partial watermarking.

4. Shows that \MethodName has minimal impact on image synthesis quality. The Fréchet Inception Distance between original and watermarked images changes by <1%.

5. Evaluates robustness of \MethodName against two countermeasures - noise injection and compression. Accuracy remains high (>74%) while countermeasures degrade image quality.

6. Discusses limitations of current cross-model transferability and uniqueness of watermarks. Suggests future improvements via model-specific fine-tuning and subject-specific generator tuning.

In summary, the key contributions are developing a new generative watermarking approach for subject-driven synthesis, extensive experimental validation of effectiveness even in challenging scenarios, and analysis of limitations to guide future work.

Method Section §

Here is a summary of the methodology section from the paper:

The key components of the proposed Generative Watermarking Against Unauthorized Subject-Driven Image Synthesis (\MethodName) are:

• A watermark generator G that adds imperceptible perturbations as watermarks to images
• A watermark detector D that distinguishes watermarked vs normal images

The pipeline has two phases:

Phase 1 - Pre-training G and D:

• G takes a latent code z to generate a watermark w of same size as the image
• w is added to the original image x to get watermarked image x_w
• D (CNN) is trained to classify images as clean or watermarked
• G is trained to minimize visibility of w measured by LPIPS distance between x and x_w
• Joint training optimizes a weighted sum of objectives for G and D

Phase 2 - Fine-tuning D for each subject:

• Train clean and watermarked subject-driven models on original and watermarked images
• Synthesize images from the models with diverse prompts
• Fine-tune D on synthesized images to improve personalized detection

This incorporates the synthesis process into watermark optimization and ensures uniqueness.

In summary, \MethodName relies on joint optimization of a generator and detector, with a fine-tuning step personalized for each subject. The key aspects are the generative adversarial framework, perceptual optimization for invisibility, and incorporation of subject-specific synthesis.

High-Level Pseudo Code §

Here is the high-level pseudo code for the key steps of the method proposed in this paper:

# Phase 1 - Pre-train watermark generator G and detector D
 
# Generate watermark w using latent code z 
w = G(z)  
 
# Add watermark to original image x
x_w = x + w   
 
# Train detector D to classify clean vs watermarked images
D_loss = BCE(D(x), 0) + BCE(D(x_w), 1)
 
# Train generator G for invisibility measured by LPIPS 
G_loss = max(LPIPS(x, x_w) - p, 0)
 
# Joint optimization with weighted objectives
loss = α*G_loss + D_loss
 
# Update G and D to minimize loss
 
# Phase 2 - Fine-tune detector D for each subject
 
# Train clean and watermarked models on original and watermarked images
M = train_model(X)  
M_w = train_model(X_w)
 
# Generate synthesized images using the models
S = M.generate_images(prompts) 
S_w = M_w.generate_images(prompts)
 
# Fine-tune detector D on synthesized images
D_fine_tune_loss = BCE(D(S), 0) + BCE(D(S_w), 1) 
 
# Update D to minimize fine-tuning loss

In summary, the key steps are:

1. Jointly pre-train a generator and detector for robustness and invisibility
2. For each subject, fine-tune the detector using synthesized images to improve personalized detection The main aspects are the generative adversarial learning and incorporation of subject-specific synthesis.

Detailed Pseudo Code §

Here is a more detailed pseudo code to implement the key aspects of the method proposed in this paper:

# Hyperparameters
p = 0.05 # Invisibility level
α = 1 # Balancing factor 
 
# Phase 1 - Pre-train watermark generator G and detector D
 
# Generator G
class Generator(nn.Module):
  # Architecture details
  ...
  
  def forward(self, z):
    return self.generate(z) # Generates watermark
  
G = Generator()
 
# Detector D  
class Detector(nn.Module):
  # Architecture details
  ...
 
  def forward(self, x):
    return self.detect(x) # Predicts watermarked or not
  
D = Detector()
 
# Generate watermark
z = torch.randn(batch_size, latent_dim) 
w = G(z)
 
# Get watermarked images
x_w = x + w 
 
# Losses
lpips_loss = max(LPIPS(x, x_w) - p, 0) 
bce_loss = BCE(D(x), 0) + BCE(D(x_w), 1)  
 
loss = α*lpips_loss + bce_loss
 
# Update G and D parameters
opt_G = torch.optim.Adam(G.parameters()) 
opt_D = torch.optim.Adam(D.parameters())
 
for i in range(num_epochs):
  opt_G.zero_grad()
  opt_D.zero_grad()
  
  l_G, l_D = loss.chunk(2)  
 
  l_G.backward()
  opt_G.step()
 
  l_D.backward()
  opt_D.step()
 
 
# Phase 2 - Fine-tune detector D for each subject
 
# Get synthesized images 
S = synth_model(X).generate(prompts) 
S_w = synth_model(X_w).generate(prompts)
 
# Fine-tune detector
for x, y in zip(S, S_w):
  loss = BCE(D(x), 0) + BCE(D(y), 1)
  
  loss.backward()
  opt_D.step()

In summary, key aspects are:

• Generator and detector architectures
• Watermark generation from latent code
• BCE and LPIPS losses for optimization
• Fine-tuning detector on subject-specific synthesized images