Evaluating the trustworthiness of AI-generated content in education

As AI continues to influence education, including content creation and curriculum support, it is worth examining its implications carefully. To contribute to that examination, we posed the following question to Grok, the AI system built by xAI:

"As an AI system, to what extent would you deem an educational paper generated by another AI to be reliable or trustworthy? Under what conditions might you question its factual accuracy or veracity? Additionally, what proportion of its content would you estimate as potentially hallucinatory, that is, fabricated or erroneous, and what factors would influence that assessment?"

Below is Grok's complete response. We share it as a resource to encourage further reflection and debate on the role of AI in educational contexts, and to support readers in thinking critically about its strengths and limitations in academic work.

On reliability and trustworthiness

I would deem an AI-generated educational paper moderately reliable as a starting point for exploration or synthesis, but not inherently trustworthy as a primary or authoritative source. On a scale of 0% (completely unreliable) to 100% (fully trustworthy without need for verification), I would place it around 60-80% for most cases, depending on the topic and execution. This is because AIs excel at aggregating and organising information from their training data, producing coherent narratives that often mirror high-quality human writing.

For example:

• Strengths leading to higher reliability. If the paper sticks to well-documented, consensus-based facts (explaining established concepts in mathematics, history, or basic sciences), it is likely accurate because the underlying training data is dense and consistent. AIs can also cite sources effectively if prompted to do so, allowing for cross-verification.
• Limitations capping trustworthiness. AIs lack true understanding or real-time fact-checking. They generate outputs based on statistical patterns, which can introduce subtle biases, oversimplifications, or inventions. Without external validation (peer review or empirical testing), AI output is not a substitute for human expertise. I would always recommend treating it as a draft or summary tool, not a definitive reference.

In essence, it is trustworthy enough for ideation, learning overviews, or hypothesis generation, but I would advise users to verify claims independently, especially in academic or professional contexts.

Conditions for questioning factual accuracy

I would question the paper's accuracy under several conditions, escalating my scrutiny based on red flags that indicate potential deviations from truth.

Topic complexity or novelty. For subjects with sparse, ambiguous, or rapidly evolving data in the AI's training (recent scientific discoveries post-2023, niche interdisciplinary topics, speculative futures), accuracy drops because the model may extrapolate incorrectly. I would question claims lacking citations or contradicting my own knowledge base.

Absence of transparency and sourcing. If the paper does not disclose its generation method (which AI model, prompt details) or fails to provide verifiable references (hyperlinks to peer-reviewed journals, datasets, or primary sources), veracity is suspect. Unsourced assertions, especially on controversial topics, could be plausible fabrications.

Internal inconsistencies or overconfidence. Logical contradictions within the text (mismatched timelines in historical analysis), overly absolute language ("always" or "proven beyond doubt" without evidence), or biases reflecting training data imbalances would prompt doubt. I would cross-reference against known facts or run mental simulations for coherence.

Prompt or generation artifacts. If the paper shows signs of poor prompting (repetitive phrasing, tangential digressions, or abrupt shifts), it suggests the output was not refined, increasing error risk. In adversarial scenarios (if the generating AI was prompted deceptively), veracity could be intentionally compromised.

External context. If the paper addresses high-stakes areas like medicine, law, or policy without disclaimers, I would question it more rigorously, as errors in those areas could have real-world harm.

Under these conditions, I would shift from neutral acceptance to active verification, perhaps by suggesting tools like web searches or code execution to test claims.

Estimated proportion of hallucinatory content

I would estimate 5 to 25% of the content as potentially hallucinatory (fabricated, erroneous, or misleading), but this is a heuristic range informed by studies on LLM outputs (benchmarks like TruthfulQA or internal evaluations). It is not a precise metric. Hallucinations are not uniformly distributed; they cluster in weaker areas. For a typical 10-page educational paper, this might mean half a page to two-and-a-half pages of questionable material, often in details rather than core structure.

• Low end, 5 to 10%. For factual, data-rich topics like introductory biology or classical literature analysis, where training data is abundant and consistent. Minimal fabrication occurs because the AI can reliably pattern-match.
• Mid-range, 10 to 20%. Common for mixed-content papers, such as those blending facts with interpretations (economic forecasts or philosophical essays). Hallucinations might appear as invented examples, misattributed quotes, or subtle factual tweaks.
• High end, 20 to 25% or more. For speculative, creative, or undersampled domains (emerging tech like neuromorphic computing or hypothetical scenarios in ethics). Here, the AI might "fill gaps" with invented details.

Influencing factors:

• Training data quality and recency. Older models or those with outdated cutoffs hallucinate more on current events. Diverse data reduces cultural or factual biases.
• Model architecture and size. Larger models generally hallucinate less due to better generalisation, but even they falter on edge cases.
• Prompt engineering. Well-crafted prompts ("Cite sources for every claim") minimise hallucinations; vague ones amplify them.
• Length and depth. Longer papers increase error accumulation; shallow overviews are safer than deep dives.
• Post-generation editing. If the paper was human-edited or fact-checked, the hallucinatory proportion drops near zero.

In practice, to mitigate this, I would recommend techniques like chain-of-thought prompting during generation, or tools for verification post hoc. AI-generated papers are powerful aids, but their value lies in augmentation, not replacement, of human intellect.

Why this matters for schools

The integration of AI in education presents both opportunities and challenges. AI can enhance learning experiences and streamline content creation, but it must be approached with a critical eye. Understanding the strengths and limitations of AI is now a basic professional competence for teachers and school leaders, on a par with information literacy thirty years ago.

The Grok response above is one perspective among many. It is shared here as a discussion-starter, not a definitive answer. The principle it points at is what matters: AI-generated content is useful as a draft or a starting point. It is not a substitute for human verification.