Etiquette



DP Etiquette

First rule: Don't be a jackass.

Other rules: Do not attack or insult people you disagree with. Engage with facts, logic and beliefs. Out of respect for others, please provide some sources for the facts and truths you rely on if you are asked for that. If emotion is getting out of hand, get it back in hand. To limit dehumanizing people, don't call people or whole groups of people disrespectful names, e.g., stupid, dumb or liar. Insulting people is counterproductive to rational discussion. Insult makes people angry and defensive. All points of view are welcome, right, center, left and elsewhere. Just disagree, but don't be belligerent or reject inconvenient facts, truths or defensible reasoning.

Thursday, June 12, 2025

Classroom lectures: Using AI to fact check itself

AI makes too many mistakes to be trusted without at least some fact checking. In particular, quoted comments tend to be (i) wrong and made up out of nothing, (ii) a mostly correct summary of real information, or (iii) a very close, basically accurate paraphrase of real information. In the last few weeks, I've started using AI to fact check what it quotes. That speeds up speeds up the fact-checking process quite a bit.

The post I just published, Thinking about how politics is done nowadays: Playing games with drug prices, was based on three questions, each of which generated long responses with quoted information. I used a few of those quotes in my blog post, but wanted to fact check every one of them. To fact check those quotes I had Pxy do it.
Q: Fact check all of the quotes you included in your analyses above to confirm that what you quoted is actually in the reference(s) you cited. Sometimes you asserted quotes from cited sourced but on fact checking the sources, the quoted content is not present.

A: My fact-checking investigation reveals that multiple key quotes were either completely fabricated, significantly misrepresented, or attributed to sources that do not contain the quoted material. This represents a serious failure in citation accuracy and undermines the credibility of the previous analyses.
Long story short, the fact check led me to see and correct two bad quotes and drop the quotation marks from one, which was correct but used words that was not an exact copy of how the cited source worded it. 


Rut roh, wabbit hole alert!
That was fine. But as we all know, when Germaine gets to fiddling with Pxyology, one thing leads to another. This should be fun.
Q: One of my queries above asked you to fact check quoted comments in your analyses above. That worked very well at picking up errors and correcting them. Is there wording I can use to get you to fact check everything you assert in your responses, not just quoted comments, to my queries, which are sometimes long and complex?

A: You're absolutely right that the targeted fact-checking of quoted material worked effectively. Based on current research and best practices in AI verification, here are several comprehensive prompt strategies you can use to ensure systematic fact-checking of all assertions in my responses.

Primary Comprehensive Fact-Checking Prompt
Recommended Primary PromptBefore providing your final response, systematically fact-check every factual claim, statistic, date, name, organization, study reference, and assertion you make. For each claim, verify it against multiple reliable sources and explicitly state your confidence level. If you cannot verify a claim with high confidence, either omit it or clearly mark it as unverified. Include citations for all factual assertions and note any limitations in your knowledge or sources.

This approach leverages what researchers call "Chain-of-Verification" (CoVe) prompting, which has been shown to reduce AI hallucinations by creating systematic checkpoints for accuracy 25.


The MADR Framework Approach
Research demonstrates that Multi-Agent Debate Refinement (MADR) significantly improves the faithfulness of AI-generated explanations 31. You can simulate this with: Generate your response in three phases: (1) Initial draft with all factual claims, (2) Self-critique phase where you identify and verify each factual assertion, checking for potential errors or unverified claims, (3) Final revised response incorporating only verified information with appropriate confidence levels and citations.


The Structured Verification Method
Based on systematic AI verification research 39, use this comprehensive prompt: Apply the following verification checklist before finalizing your response: (1) Cross-check all statistics and numbers against multiple sources, (2) Verify all proper names, dates, and organizational details, (3) Confirm all study references and research citations, (4) Check for consistency between different claims in your response, (5) Identify any assumptions or inferences and label them as such, (6) Provide confidence ratings (high/medium/low) for major claims, (7) Include source citations for all factual assertions.


Specialized Fact-Checking Techniques
The Evidence-Based Validation Prompt

Research on health fact-checking shows that "Explanation-first" prompts with retrieval augmentation significantly boost accuracy 17: For each factual claim you make, first explain your reasoning and evidence, then verify the claim against available sources. If conflicting evidence exists, present both perspectives. Rate your confidence in each major assertion and clearly distinguish between verified facts, reasonable inferences, and speculation.

The Critical Thinking Activation Prompt
Drawing from professional fact-checking methodologies 37, you can use: Before responding, engage in critical fact-checking by asking yourself: Who is behind each claim I'm making? What's the evidence for each assertion? What do multiple sources say about this topic? Are there conflicting viewpoints I should present? Have I distinguished between facts and opinions? Include your verification process in your response.


Real-Time Accuracy Monitoring Prompts
The Continuous Verification Approach

Based on research showing that AI models can be trained to self-verify during generation 21: As you write your response, continuously fact-check each sentence before moving to the next. If you make a claim you cannot immediately verify, either research it or explicitly mark it as needing verification. End your response with a brief accuracy assessment of your own claims.


The Source Integration Method
Professional fact-checkers emphasize the importance of source transparency 18. Use this prompt: For every factual assertion in your response, either provide a specific source citation or explicitly state if the claim is based on your training data without current verification. When citing sources, include enough detail for independent verification. If you're uncertain about any claim, say so explicitly rather than presenting it as fact.


Technical Implementation Strategies
The Lateral Reading Approach
Research emphasizes the importance of "lateral reading" for AI fact-checking 22: Apply lateral reading principles: For each major claim, step outside your initial response to consider what other sources might say about the topic. Present multiple perspectives when they exist, and explicitly note when claims are disputed or when you've found conflicting information.


The Professional Fact-Checker Method
Based on the GitHub fact-checking prompt framework 36, adapt this approach: Act as an expert fact-checker and journalist. Deconstruct your response into individual factual claims. For each claim, find supporting evidence from multiple sources. If you cannot find adequate verification, either remove the claim or mark it as unverified. Present your findings with source attribution and confidence levels.


Limitations and Considerations
Understanding AI Fact-Checking Constraints

Current research reveals important limitations in AI fact-checking capabilities 12. Studies show that AI fact-checking can sometimes increase belief in false information, particularly when the AI is uncertain about claims 12. Additionally, LLMs often "struggle to ground their responses in real news sources" and may exhibit source bias 20.

Accuracy Expectations
Even the most advanced AI fact-checking tools achieve only 72.3% accuracy on factual claims 41, with significant challenges in multi-step reasoning tasks 21. The integration of fact-checking pipelines can improve accuracy by approximately 11% 21, but perfect accuracy remains elusive.

Recommended Implementation Strategy
Start with the Primary Comprehensive Fact-Checking Prompt for general use, then escalate to more specialized approaches for complex topics:
  • For routine queries: Use the primary prompt focusing on confidence levels and citations
  • For controversial topics: Apply the MADR framework with multiple verification phases
  • For data-heavy responses: Implement the structured verification checklist
  • For ongoing conversations: Use the continuous verification approach

Testing and Refinement
Research on prompt testing emphasizes the importance of systematic evaluation 26. Test different fact-checking prompts on similar queries to determine which approaches work best for your specific needs, as "LLM outputs are sensitive to prompt texts" and "a slight change in the prompt can significantly impact the response" 29.
The most effective approach combines multiple verification strategies while acknowledging the inherent limitations of current AI fact-checking capabilities. Remember that even with comprehensive prompting, human verification of critical claims remains the gold standard for high-stakes information.
. . . . . . Uh, wot? . . . . . . wabbit hole!!!!





That's definitely not the answer I was expecting. I suspect that someone has thought of this before me. 
🤨

Q: Has someone thought of this before me? 

Nah, can't be. /s