Notes on T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation

Link to paper: https://arxiv.org/abs/2307.06350

Paper published on: 2023-07-12

Paper's authors: Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, Xihui Liu

Let's walk through a new frontier in the field of text-to-image generation. Imagine a painter who can create intricate scenes from your verbal descriptions, but sometimes struggles with the complexity of the composition. This is akin to current text-to-image models, which have difficulty composing objects with different attributes and relationships into a coherent scene. To address this, the research paper we're discussing today introduces T2I-CompBench, a comprehensive benchmark for open-world compositional text-to-image generation, and a new approach called GORS to enhance the compositional abilities of these models.

T2I-CompBench is like a rigorous training course for our painter – it consists of 6,000 compositional text prompts divided into three categories: attribute binding, object relationships, and complex compositions, further divided into six sub-categories. This benchmark gives us a diverse range of scenarios to test and improve our models.

To measure the performance of our models on this benchmark, the researchers propose several evaluation metrics, much like a grading system. These include Disentangled BLIP-VQA for attribute binding, UniDet-based spatial relationship evaluation, and a 3-in-1 metric for complex compositions. These metrics are designed to evaluate different aspects of compositionality, such as color, shape, texture, and object relationships.

Now, let's talk about the GORS (Generative Object Relationship Synthesis) approach. It's like a personal trainer for our models, fine-tuning them with reward-driven sample selection to improve their compositional ability. This approach involves fine-tuning both the CLIP text encoder and the U-Net of Stable Diffusion with LoRA, which is more effective than fine-tuning either component separately.

The effectiveness of this approach is evident when we examine the performance of different models. For instance, Stable Diffusion v2 consistently outperforms its previous versions, and Attend-and-Excite built upon Stable Diffusion v2 shows improved performance in attribute binding. Interestingly, Composable Diffusion, also built upon Stable Diffusion v2, doesn't perform as well. However, the GORS approach outperforms all previous approaches across all types of compositional prompts.

The study also reveals that spatial relationships are the hardest for text-to-image models to grasp, while non-spatial relationships are the easiest. This is akin to our painter finding it easier to paint objects independently rather than in relation to each other.

Setting a threshold for selecting samples for fine-tuning is crucial, as using a half threshold or zero threshold leads to worse performance. This is like our painter choosing the right amount of paint – too little or too much can ruin the artwork.

The researchers also conducted a human evaluation on Amazon Mechanical Turk, where each image-text pair was rated by three human annotators on a scale of 1 to 5 for image-text alignment. The proposed evaluation metrics showed significant improvement over existing metrics in terms of correlation with these human evaluation scores.

The T2I-CompBench dataset construction includes prompts for color, shape, texture, non-spatial relation, and complex compositions. Meanwhile, the MiniGPT4-CoT evaluation metric includes prompts for attribute binding, spatial relationships, non-spatial relationships, and complex compositions.

However, the research is not without limitations. The model's performance is slightly lower on unseen attribute-object combinations compared to seen combinations. Also, the MiniGPT-4 evaluation without Chain-of-Thought does not align strictly with human evaluation results.

The study also acknowledges potential negative social impacts, such as the abuse of text-to-image models and biases in hallucinations and generated content. This is an important reminder that while we strive to improve our painter's skills, we must also consider the ethical implications of our work.

In conclusion, this research provides a robust benchmark and novel approaches for improving the compositional abilities of text-to-image models. The proposed T2I-CompBench and GORS approach, along with the new evaluation metrics, provide a comprehensive toolkit for advancing this field. As we continue to explore this exciting domain, we're getting closer to the day when our painter can flawlessly bring any description to life.