Operational Trade-offs in LLM-Based Misinformation Classification¶
This is a research program done in Pioneer Academics Program (2026 Spring).
The paper evaluates whether adding the complexity of an LLM inference system improves misinformation classification on finance and healthcare datasets. We compare Single-Shot, same-prompt majority Voting, and a role-based Mixture of Agents architectures with retrieval-augmented generation as the optional layer.
Read the paper (Yicheng Qiu, 2026).
All experiment design, datasets, models, and results are described in the paper. Experiment source code is hosted on GitHub.
Abstract¶
With the rise of social media platforms, the volume of unverified information has grown faster than the human capacity to check it. An automated information verification system can reduce the human workload, and the strong reasoning power of Large Language Models (LLMs) makes them a promising tool for fact-checking. In this paper, we evaluate Single-Shot, Same-Prompt Majority Voting, and a role-based Mixture of Agents architectures, each with and without Retrieval-Augmented Generation (RAG), on 500 claims each in finance and healthcare. Each architecture generates a final label of REAL, FAKE, and ESCALATE. The performance is analyzed using metrics of F1, precision, recall, false positive rate, ESCALATE rate, latency, and computational cost. The experiments suggest an adaptation strategy based on operational purposes. The paper also analyzes the limitations of adding architectural complexity to LLM stacks and of introducing mixed-quality retrieved information. Voting with RAG achieves the highest observed finance F1 (0.803) but escalates 43.2% of entries, while Voting without RAG achieves the highest observed healthcare F1 (0.794), but 0.015 above Single-Shot. Among the tested configurations, Single-Shot offered the strongest overall trade-off among computational cost, ESCALATE rate, latency, and false-positive rate. The effect of the tested RAG design differed across the datasets and model configurations.