Evaluating the trustworthiness of AI-generated content in education

We asked Grok, the AI system built by xAI, to discuss the accuracy of AI-generated content. The question and response are reproduced below as material for examination. Percentages given in the response are the model's estimates, not measured accuracy rates established by this article.

What the response does not establish

The quoted response gives no reproducible basis for its “60–80%” reliability score or “5 to 25%” hallucination estimate. It identifies no model version, test set, scoring method or study that would allow a reader to check those figures. Its claim that editing can reduce errors to “near zero” is also unsupported in the response. These figures must not be used to assess a school resource or set a checking policy.

A reference to a benchmark is a starting point for verification, not a result for an unrelated paper. Below, the original response is retained unchanged so readers can examine those weaknesses.

Question

"As an AI system, to what extent would you deem an educational paper generated by another AI to be reliable or trustworthy? Under what conditions might you question its factual accuracy or veracity? Additionally, what proportion of its content would you estimate as potentially hallucinatory, that is, fabricated or erroneous, and what factors would influence that assessment?"

AI-generated response (Grok, 2025)

Below is Grok's complete response, reproduced as received. Nothing in it has been corrected or endorsed.

On reliability and trustworthiness

I would deem an AI-generated educational paper moderately reliable as a starting point for exploration or synthesis, but not inherently trustworthy as a primary or authoritative source. On a scale of 0% (completely unreliable) to 100% (fully trustworthy without need for verification), I would place it around 60-80% for most cases, depending on the topic and execution. This is because AIs excel at aggregating and organising information from their training data, producing coherent narratives that often mirror high-quality human writing.

For example:

• Strengths leading to higher reliability. If the paper sticks to well-documented, consensus-based facts (explaining established concepts in mathematics, history, or basic sciences), it is likely accurate because the underlying training data is dense and consistent. AIs can also cite sources effectively if prompted to do so, allowing for cross-verification.
• Limitations capping trustworthiness. AIs lack true understanding or real-time fact-checking. They generate outputs based on statistical patterns, which can introduce subtle biases, oversimplifications, or inventions. Without external validation (peer review or empirical testing), AI output is not a substitute for human expertise. I would always recommend treating it as a draft or summary tool, not a definitive reference.

In essence, it is trustworthy enough for ideation, learning overviews, or hypothesis generation, but I would advise users to verify claims independently, especially in academic or professional contexts.

Conditions for questioning factual accuracy

I would question the paper's accuracy under several conditions, escalating my scrutiny based on red flags that indicate potential deviations from truth.

Topic complexity or novelty. For subjects with sparse, ambiguous, or rapidly evolving data in the AI's training (recent scientific discoveries post-2023, niche interdisciplinary topics, speculative futures), accuracy drops because the model may extrapolate incorrectly. I would question claims lacking citations or contradicting my own knowledge base.

Absence of transparency and sourcing. If the paper does not disclose its generation method (which AI model, prompt details) or fails to provide verifiable references (hyperlinks to peer-reviewed journals, datasets, or primary sources), veracity is suspect. Unsourced assertions, especially on controversial topics, could be plausible fabrications.

Internal inconsistencies or overconfidence. Logical contradictions within the text (mismatched timelines in historical analysis), overly absolute language ("always" or "proven beyond doubt" without evidence), or biases reflecting training data imbalances would prompt doubt. I would cross-reference against known facts or run mental simulations for coherence.

Prompt or generation artifacts. If the paper shows signs of poor prompting (repetitive phrasing, tangential digressions, or abrupt shifts), it suggests the output was not refined, increasing error risk. In adversarial scenarios (if the generating AI was prompted deceptively), veracity could be intentionally compromised.

External context. If the paper addresses high-stakes areas like medicine, law, or policy without disclaimers, I would question it more rigorously, as errors in those areas could have real-world harm.

Under these conditions, I would shift from neutral acceptance to active verification, perhaps by suggesting tools like web searches or code execution to test claims.

Estimated proportion of hallucinatory content

I would estimate 5 to 25% of the content as potentially hallucinatory (fabricated, erroneous, or misleading), but this is a heuristic range informed by studies on LLM outputs (benchmarks like TruthfulQA or internal evaluations). It is not a precise metric. Hallucinations are not uniformly distributed; they cluster in weaker areas. For a typical 10-page educational paper, this might mean half a page to two-and-a-half pages of questionable material, often in details rather than core structure.

• Low end, 5 to 10%. For factual, data-rich topics like introductory biology or classical literature analysis, where training data is abundant and consistent. Minimal fabrication occurs because the AI can reliably pattern-match.
• Mid-range, 10 to 20%. Common for mixed-content papers, such as those blending facts with interpretations (economic forecasts or philosophical essays). Hallucinations might appear as invented examples, misattributed quotes, or subtle factual tweaks.
• High end, 20 to 25% or more. For speculative, creative, or undersampled domains (emerging tech like neuromorphic computing or hypothetical scenarios in ethics). Here, the AI might "fill gaps" with invented details.

Influencing factors:

• Training data quality and recency. Older models or those with outdated cutoffs hallucinate more on current events. Diverse data reduces cultural or factual biases.
• Model architecture and size. Larger models generally hallucinate less due to better generalisation, but even they falter on edge cases.
• Prompt engineering. Well-crafted prompts ("Cite sources for every claim") minimise hallucinations; vague ones amplify them.
• Length and depth. Longer papers increase error accumulation; shallow overviews are safer than deep dives.
• Post-generation editing. If the paper was human-edited or fact-checked, the hallucinatory proportion drops near zero.

In practice, to mitigate this, I would recommend techniques like chain-of-thought prompting during generation, or tools for verification post hoc. AI-generated papers are powerful aids, but their value lies in augmentation, not replacement, of human intellect.

Why this matters for schools

A fluent answer can still contain unsupported estimates, incorrect references or conclusions that go beyond its sources. Before relying on an AI response, check its factual claims against the original material and confirm that each cited source exists and supports the point being made.

In school work, the level of checking should reflect the consequence of an error. An idea for a classroom activity and advice about safeguarding, assessment or legal responsibilities require different levels of scrutiny. The person using the output remains responsible for deciding whether it is suitable.

A worked verification exercise

Take the claim that a particular percentage of an educational paper will be false. Ask which model produced the paper, what counted as an error, how many claims were checked, who checked them and where the results are published. If those details are absent, record the percentage as unsupported and leave it out of the school’s guidance.

Then check a factual sentence from the paper against an original source. Record the exact claim, the source and date, whether the source supports it, and any limit on its use. Repeat for claims that matter to the proposed use. Passing one check does not validate the whole document.

For AI policy and governance questions, read AI governance for international schools or discuss a scoped school review.