Research case
Questioning the survey responses of LLMs (Dominguez-Olmedo et al., NeurIPS 2024)
Put American Community Survey questions to 43 language models and checked whether their answers survive simple controls such as shuffling answer order.
What is reported
Answers were driven by ordering and labeling biases (e.g. favoring option “A”); after randomizing order, models drifted toward uniformly random responses regardless of size.
Before you use it
Always randomize option order when probing a model with survey items, and report it.
Sources and review
Source checked · Not tested by this site
Checked the source record and abstract. Results describe the reported study, not an independent replication by this site.
License and reuse
No license has been recorded. Check the source’s license before reusing code, models or data.
A license note is not a license audit. Open source does not guarantee free hosting, model access or commercial reuse.