5th Year M.S. Thesis Presentation - Siddharth Parekh
July 28, 2026 2:00PM—3:20PM
Location:
In Person
-
Tepper Building 1403
Speaker:
SIDDHARTH PAREKH,
Master's Student, Computer Science Department, Carnegie Mellon University
https://sparekh.me/
A table is a two-dimensional object, but a decoder-only language model reads it as a one-dimensional stream of tokens. This thesis asks how such models recover the row-column structure needed to answer a question about a single cell, and argues that the computation splits into two stages: an earlier matching stage that binds the queried column and row to landmarks in the serialised table, and a later retrieval stage that locks onto the target cell. Cell lookup is the case in which this addressing becomes the task and can be scored in isolation; every richer query over a flat table must address a value before it can compute with one.
Behavioural and causal analysis of two open-weight model families (Llama-3.1-8B, Qwen-2.5-7B) on real tables converges onto the same picture. The two-stage ordering is robust, and it persists in the conditions where the model fails - so the formatting that makes tables easy to read feeds the matching stage rather than creating it. Intervention separates what the model needs from what carries the answer: matching is necessary but not sufficient alone, the key heads matter as a group more than individually, and the two axes break independently, the signature of genuine coordinate addressing rather than one undifferentiated match. The faithful circuit is a small fraction of the model yet much larger than a hand-picked roster of heads, and the two families distribute the same roles differently. Column matching keys on cell content rather than on delimiters, but does so lexically, which bounds the account to keys appearing verbatim in the prompt.
Because each stage depends on a particular property of the prompt, this picture converts into formatting guidance: an explicit row-index column turns row lookup from positional tracking into token matching and removes the dominant failure mode; markdown outperforms CSV, and narrow tables, explicit headers, and question wording that matches the table each protect a stage that would otherwise break down. These are explanations rather than heuristics, since each corresponds to an operation the model has to perform.
Additional Information
Thesis Committee
Carolyn Rose (Chair)
Daphne Ippolito
For More Information:
amalloy@cs.cmu.edu