Doctoral Thesis Oral Defense - Joshua Williams

— 12:00pm

Location:
In Person and Virtual - ET - Reddy Conference Room, Gates Hillman 4405 and Zoom

Speaker:
JOSHUA NATHANIEL WILLIAMS , Ph.D. Candidate
Computer Science Department
Carnegie Mellon University
https://jnwilliams.github.io/

Understanding Representations of Humans in Generative Image Modeling Through Discrete Counterfactual Prompt Optimization

Text-to-image (T2I) models are widely used generative systems, making it essential to understand how they represent human subjects. Comparing generated images across carefully designed prompts can reveal representational patterns, some of which reflect harmful biases requiring intervention. Existing approaches often rely on fixed prompt templates or identity categories, which are useful for benchmarking but risk blind spots shaped by researchers’ assumptions.

This thesis introduces methods grounded in counterfactual and contrastive analysis to uncover representational asymmetries and harms beyond predefined categories. We show that effective explanations for classifiers must account for the underlying data distribution; without this, analyses risk spurious conclusions. To address this, we adapt the graphical model underlying counterfactual explainability and propose a new distribution-aware metric.

Building on these insights, we further develop distributionally informed approaches to prompt optimization in T2I settings. Our framework incorporates multiobjective optimization across language models with distinct tokenizers and embeddings, enabling richer exploration of representational behaviors. Finally, we present an unsupervised strategy for surfacing candidate prompts that reveal previously undocumented asymmetries. By linking the linguistic patterns of generative models to their visual outputs, we advance methods for diagnosing biases and targeting specific representational behaviors in training and evaluation.  

Thesis Committee
Zico Kolter (Chair)
Hoda Heidari
Aditi Raghunathan
Sarah Laszlo (Visa)

In Person and Zoom Participation.  See announcement.

For More Information:
matthewstewart@cmu.edu

Add event to Google
Add event to iCal