Notes on Llama 2: Open Foundation and Fine-Tuned Chat Models

Link to paper: https://arxiv.org/abs/2307.09288

Paper published on: 2023-07-18

Paper's authors: Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, Thomas Scialom

GPT3 API Cost: $0.16

GPT4 API Cost: $0.21

Total Cost To Write This: $0.37

Time Savings: 70:1

The ELI5 TLDR:

Scientists have created two new AI models called Llama 2 and Llama 2-Chat. These models are really good at having conversations and can give better answers than other chat models. They were trained using a lot of data and special techniques to make them safe and helpful. The Llama 2-Chat model is open-source, which means more people can use it. The scientists also made sure to test the models to make sure they are safe and not mean. They want people to use the models responsibly and follow the rules for using them.

The Deeper Dive:

Summary: The Evolution of Llama 2 and Llama 2-Chat Models

In this cutting-edge AI research, the authors introduce Llama 2 and Llama 2-Chat, two large language models (LLMs) that have been fine-tuned and optimized for dialogue use cases. These models outperform open-source chat models in most benchmarks, making them suitable substitutes for closed-source models. The researchers also highlight their approach to fine-tuning these models and the steps taken to enhance their safety, including specific data annotation and tuning, red-teaming, and iterative evaluations.

Imagine you're trying to build a chatbot for customer service. The Llama 2-Chat model could be a game-changer, providing more accurate and safer responses, while also being open-source and therefore more accessible.

The Making of Llama 2 Models

Llama 2 models were created using an optimized auto-regressive transformer with several enhancements to improve performance. The training corpus consists of a mix of data from publicly available sources, excluding data from Meta's products or services. The models were trained on a whopping 2 trillion tokens of data, with factual sources up-sampled to increase knowledge and reduce hallucinations.

The models use the standard transformer architecture, pre-normalization, a SwiGLU activation function, and rotary positional embeddings. They also have an increased context length and use grouped-query attention for improved inference scalability. The training was conducted on Meta's Research Super Cluster and internal production clusters using NVIDIA A100 GPUs.

The Performance of Llama 2 Models

The Llama 2 models were evaluated on various benchmarks, including code, commonsense reasoning, world knowledge, and reading comprehension. They showed improved performance compared to Llama 1 and Falcon models on most benchmarks. The Llama 2 70B model, in particular, outperforms all open-source models and is on par or better than the PaLM model on most benchmarks.

The Creation of Llama 2-Chat

The Llama 2-Chat model is the result of several months of research and iterative applications of alignment techniques. Supervised Fine-Tuning (SFT) was used to train Llama 2-Chat. It was fine-tuned for 2 epochs using a cosine learning rate schedule. After SFT, Reinforcement Learning with Human Feedback (RLHF) was applied to align Llama 2-Chat with human preferences.

Human preference data was collected for reward modeling. The reward model was trained to output a scalar score indicating the quality of model generations. Llama 2-Chat was optimized during RLHF for better human preference alignment and improved helpfulness and safety.

The Safety Measures

Safety is a critical concern when it comes to AI models. The researchers trained two separate reward models, one optimized for helpfulness and another for safety. The reward models were trained using collected pairwise human preference data and were fine-tuned using Proximal Policy Optimization (PPO) and Rejection Sampling algorithms.

The researchers also introduced Ghost Attention (GAtt) to improve multi-turn consistency in dialogue systems. GAtt helps the attention focus in multi-turn conversations by synthetically concatenating an instruction to user messages and fine-tuning the model based on this augmented data.

Evaluation and Safety Testing

Model-based evaluation is challenging, and human evaluation is considered the gold standard but not always scalable. Llama 2-Chat models outperform open-source models on helpfulness prompts. The largest Llama 2-Chat model is competitive with ChatGPT.

Safety investigations were conducted on pretraining data and pretrained models, including analysis of language distributions, demographic representations, toxicity, and testing on existing safety benchmarks. Steps were taken to pretrain responsibly, including privacy and legal review processes, exclusion of certain data sources, and efforts to reduce the carbon footprint of pretraining.

Responsible Use of Llama 2 Models

Despite the impressive performance of Llama 2 models, they should be used with caution and only deployed after significant safety tuning. The training data for Llama 2 models shows biases in terms of gender pronouns and demographic identities. The pretraining data contains a small amount of toxicity.

The researchers use supervised safety fine-tuning to align the model with safety guidelines before RLHF. They integrate safety into the RLHF pipeline by training a safety-specific reward model and gathering challenging adversarial prompts. They refine the RLHF pipeline with safety context distillation, which involves generating safer model responses by adding a safety preprompt and fine-tuning the model on the safer responses.

Conclusion

The researchers have provided code examples to help developers replicate safe generations with Llama 2-Chat and apply basic safety techniques. The release of Llama 2 is part of an open approach to encourage responsible AI innovation and collaboration with the AI community. However, users of the pretrained models should be cautious and follow the Responsible Use Guide for tuning and deployment.

This research has the potential to revolutionize how we use AI in dialogue systems, providing safer and more accurate responses. However, it's essential to remember that while the models have been improved, they are not perfect and should be used responsibly.