Mingjie Sun

Hidden Properties of Large Language Models Degree Type: CS
Advisor(s): Elaine Shi, Guilia Fanti
Graduated: May 2025

Abstract:

Large Language Models (LLMs) are deep learning models trained to understand and generate natural language. Over the course of my PhD, LLMs have profoundly transformed the field of machine learning. They are being actively deployed into numerous commercial products, such as ChatGPT. Moreover, the principles and experiences learned from developing LLMs are still shaping the landscape of machine learning research through paradigms like scaling laws and self-supervised representation learning. However, these rapid advancements may also obscure many fundamental questions about their internal mechanisms and behaviors. As LLM capabilities grow, rigorous scientific investigation beyond conventional training and evaluation workflow is crucial for deeper understanding and continued improvement.

This thesis investigates previously overlooked 'hidden properties' of LLMs. These hidden properties span the internal weight and activation spaces, as well as their output behaviors. First, we show that LLMs are intrinsically sparse in their weight space. To demonstrate this hidden property, we develop a principled pruning approach that is able to extract effective, sparse sub-networks from pretrained models. Next, we explore the activation space and reveal the existence of structured outliers in LLMs. These activations are extremely few in numbers but exceptionally high in their absolute magnitudes. We call them massive activations. We show that these activations are closely tied to the self-attention mechanism, and propose an alternative attention formulation that is free from such outliers. Finally, we turn to the output space and design a conceptually simple framework to evaluate and study the existence of idiosyncrasies in LLM-generated text. We show that outputs from different models can be distinguished with remarkably high accuracies, and further characterize the specific signatures that underlie these differences. Overall, we hope this thesis can provide an alternative perspective on modern foundation models.

Thesis Committee:
Elaine Shi (Co-Chair)
Guilia Fanti (Co-Chair)
Bryan Parno
David J. Wu (University of Texas at Austin)

Srinivasan Seshan, Head, Computer Science Department
Martial Hebert, Dean, School of Computer Science

Keywords:
Deep learning, large language models