MisCC: Misinformation detection on counterfactual claims

Counterfactual claims (CCs) play an important role in everyday conversations, but they are difficult to fact-check automatically and understudied in fact-checking research. We investigated the nuances of misinformation in CCs, considering various potential truth values for antecedents and consequents, as well as causality. Through logic-based analysis, we present a logical theory comprising an algorithm for detecting misinformation in CCs and an approach for ensuring that the result is free of inconsistencies when needed. Finally, we propose a pipeline that integrates large language models (LLMs) with our theory for fact-checking CCs. In this approach, LLMs assist in interpreting natural language and making initial predictions based on their knowledge. Subsequently, our theory verifies the consistency of LLMs’ initial predictions as well as computes CCs’ truth values. We also created and released a comprehensive dataset of CCs, which supports multiple CC-related tasks, including classifying 1) CCs, 2) the verifiability of a statement 3) the truth value of a statement and 4) the truth values of CCs. Then GPT4 and Llama3 with zero-shot were applied as the baseline. Our approach improves their F1 scores for classifying CC’s false class, the misinformation, from 0.55 to 0.64 and 0.46 to 0.61, respectively. Notably, the inconsistency checker only benefits the prediction of true classes. The large portion of false CCs in our dataset represents the portion of misinformation in CCs on social media and the poor performance of GPT4 and Llama3 in classifying the false class highlights the need for more research on fact-checking CCs.