Manifund team member Caroline Ellison: bag-of-words nearly matches LLM on funding prediction, but she still prefers LLM review
Caroline Ellison's team used Claude Opus 5.5 to evaluate 530 Manifund grant proposals. The model's funding-probability predictions reached AUC 0.90, but a simple bag-of-words logistic regression also achieved 0.88. Despite the close predictive performance, Ellison argues that LLM review retains advantages in judgment quality, resistance to gaming, and explainability, and recommends continuing to use it.
ImportanceMaterialEvidenceE3 inspectableWrite-upStandard
A bag-of-words model nearly caught up with a large language model on predicting grant proposal outcomes, yet Caroline Ellison, a member of the Manifund team, still thinks LLMs should be used for review. The reasoning behind that preference is more worth understanding than the AUC numbers themselves.
Ellison's team used Claude Opus 5.5 to read all 530 proposals submitted to Manifund since July 2026—the start date was chosen to avoid proposals that might already be in the model's training data. For each proposal, they asked the model to produce three outputs: a short review, a quality score from 1 to 5, and a predicted probability that the proposal would reach its minimum funding target on Manifund.
On the subset of proposals that had already closed and whose text did not reveal whether they were funded, the Opus probability-prediction model achieved AUC 0.90 under 5-fold cross-validation. As a control, a bag-of-words logistic regression model reached 0.88 on the same dataset. The gap between the two is only 0.02, far smaller than the performance chasm typically invoked to justify complex models.
This result led Ellison to acknowledge that it does call into question how sophisticated Opus's judgments really are, but she still gave three reasons for sticking with LLM evaluation. First, even with similar predictive power, the bag-of-words model's judgments struck her as qualitatively worse—the funded proposals where bag-of-words scored highest above Opus read to her as not very strong; not that the projects themselves were bad, but often these were proposals where the author had clearly discussed funding offline before posting, and she would not have wanted to fund them based on the proposal text alone. Second, bag-of-words is obviously quite vulnerable to gaming: a proposer could score highly simply by starting with 'I don't have a LessWrong account, have never been accepted by MATS, and don't do interpretability research.' Third, LLMs can also give reasons that people can evaluate, and those reasons will probably get better over time.