The Hallucination Blind Spot: How China's 249M AI Users Became Perfect Prey for a Deception Epidemic

On May 17, 2025, two reports dropped within hours of each other. One from Shanghai Jiao Tong University revealed that 91.5% of Chinese AI users cannot reliably detect when a large language model is hallucinating. The other, from cybersecurity giant Qianxin, disclosed that over 70 provincial and municipal governments had deployed DeepSeek—with 90% of those servers running completely unprotected. Welcome to the most dangerous phase of China's AI revolution: the moment when trust outpaces verification.

Executive Summary: When Trust Becomes a Weapon

China's generative AI adoption has reached a scale unmatched by any nation: 249 million users—17.7% of the population—now interact with large language models regularly. But beneath this milestone lies a cognitive crisis that security experts, policymakers, and the AI companies themselves have barely begun to address.

*Table 1: The dual crisis of AI hallucination blindness and deepfake fraud in China, May 2025. Currency conversions at ¥7.2/$1.*

The convergence of these trends creates something unprecedented: a population that simultaneously trusts AI too much to detect its errors, while criminals exploit that trust at industrial scale. This isn't a theoretical risk. It's happening now—and the numbers suggest it's accelerating faster than any countermeasure.

The Shanghai Jiao Tong Study: Numbers That Should Terrify Policymakers

On May 17, 2025—World Telecommunication and Information Society Day—the Fourth Shanghai Science Communication Conference released what may be the most important AI public-health study you've never heard of. Led by Academician Ding Kuiling of the Chinese Academy of Sciences and Professor Xu Jian of Shanghai Jiao Tong University's School of Media and Communication, the research tracked three waves of surveys from 2023 to 2025, with the most recent wave collecting 811 stratified samples across 31 Chinese provinces in April 2025.

The findings are stark.

The Hallucination Perception Gap

Researchers used a 7-point Likert scale to measure "perceived AI hallucination" among users. A score of 4 represents neutral awareness. The results:

*Table 2: Public awareness of AI hallucinations in China (n=811, April 2025). The mean score was 3.3—well below the neutral threshold of 4.*

The mean perception score of 3.3 means the average Chinese AI user operates below neutral awareness. Nearly three in ten users are essentially blind to AI hallucinations. Another 45.6% have only vague concerns without concrete understanding of what could go wrong.

This matters because hallucinations aren't rare edge cases. They're systematic features of large language models—from arithmetic errors and logical contradictions to completely fabricated citations, historical events, and scientific claims. When 249 million people interact with tools they don't understand can lie, the cumulative misinformation potential is staggering.

The Counterintuitive Literacy Paradox

Here's where the study gets truly disturbing. Researchers found that the more users interact with AI, the less likely they are to detect its hallucinations.

*Table 3: The AI literacy paradox—familiarity breeds complacency, not competence. Source: Shanghai Jiao Tong University survey wave 3.*

The mechanism is psychological: repeated successful interactions create a "trust anchor." Users who get good results 95% of the time stop verifying the 5% where the model errs. The study also found that higher-educated users and those with stronger self-assessed AI literacy showed lower hallucination detection rates—precisely because their confidence outpaced their actual understanding of model limitations.

Professor Xu Jian, the study's lead author, summarized the finding bluntly: *"The public's enthusiasm for large models is high, but their vigilance against the negative effects of widespread AI application has not kept pace."*

Why the "Value Neutral" Myth Makes Users Vulnerable

The Shanghai Jiao Tong study uncovered a second cognitive trap with profound security implications: the widespread belief that AI is "value neutral."

On a 7-point scale measuring agreement with the statement "AI is value-neutral," 72.4% of respondents scored above 5. The modal response was 6 out of 7—meaning most users genuinely believe AI systems don't have biases, agendas, or built-in distortions.

The Psychology of Neutral Trust

The researchers traced a specific cognitive pathway:

1. User assumes neutrality: "AI is just math—math doesn't lie"

2. Trust threshold drops: Content from a "neutral" source bypasses normal skepticism

3. Verification stops: Why fact-check something that has no reason to deceive?

4. Hallucination becomes invisible: False information is consumed as true information

This isn't just academic. The study found that users who scored highest on "value neutrality" were statistically significantly less likely to detect hallucinations—even when presented with obvious factual errors.

*Table 4: The neutrality-trust trap. Users who believe AI is "neutral" are 7.5× less likely to catch hallucinations.*

The irony is profound: AI systems are anything but neutral. They're trained on internet text containing every human bias, optimized through reinforcement learning from human feedback that encodes cultural values, and deployed by companies with commercial and political incentives. But the public perception of neutrality makes these embedded biases and errors invisible—exactly when they matter most.

From Hallucination to Criminal Weapon: The Deepfake Explosion

If academic hallucinations represent the "soft" risk of AI blindness, deepfake-enabled fraud is the hard edge—and it's growing at a pace that makes the hallucination crisis look manageable.

The Fraud Data: A National Emergency

China's Ministry of Public Security has been tracking the evolution of AI-powered fraud with increasing alarm:

*Table 5: AI-powered fraud escalation in China. The 3,000% growth figure represents the explosion from near-zero AI-assisted fraud in 2021 to widespread deployment by 2023.*

The ¥9.9 price point is perhaps the most chilling detail. For the cost of a cup of coffee, anyone can purchase a custom deepfake video on Chinese e-commerce platforms. The technology has moved from state-level disinformation operations to democratized criminal infrastructure.

The Anatomy of a Modern AI Scam

The March 2025 case of Mr. Guo, a technology company executive in Fujian, illustrates the new reality. During a video call with someone who appeared to be a close friend—same face, same voice, same mannerisms—Guo transferred ¥4.3 million in under 10 minutes. The "friend" was an AI synthesis. The real friend knew nothing.

This isn't science fiction. The tools are open-source (DeepFaceLab, FaceSwap), the tutorials are on every video platform, and the computing power required is within reach of any modern laptop.

*Table 6: The industrialized pipeline of AI face-swap fraud. Each stage has been commoditized, with services available on dark web markets and even surface e-commerce platforms.*

The Qianxin research team, led by industry security research director Pei Zhiyong, has been tracking this evolution: *"Telecom fraud is showing new characteristics and new situations. Criminals are using the latest technologies—malicious editing, face swapping, voice changing, building fake 'camera environments'—to forge video content, confuse judgment, and trick victims."*

Government-Scale Deployment With 90% of Servers "Naked"

The hallucination and fraud crises aren't happening in a vacuum. They're unfolding against the backdrop of the most rapid government AI deployment in history—and the security architecture is nowhere near ready.

The DeepSeek Government Wave

At the 2025 Data Security Development Conference in Wenzhou, Qianxin Chairman Qi Xiangdong revealed a staggering figure: over 70 provincial and municipal governments across China had deployed DeepSeek for administrative use. From smart governance to finance, transportation, and manufacturing, AI large models were becoming the operational backbone of Chinese bureaucracy.

But Qi delivered a second, darker number: 90% of privatized DeepSeek deployments were completely unprotected—what cybersecurity professionals call "running naked."

What "Naked" Means in Practice

Qianxin's technical scanning of publicly accessible DeepSeek installations found:

*Table 7: Security posture of privatized DeepSeek deployments (n=8,971 Ollama servers scanned). Source: Qianxin Asset Mapping Eagle Platform, May 2025.*

The implications extend beyond data breaches. An unprotected government AI model can be:

- Prompt-injected to generate false official-sounding documents

- Data-poisoned through adversarial training inputs

- Used as a relay for disinformation campaigns at machine scale

- Manipulated to produce harmful or policy-subverting content

Qi Xiangdong's assessment at the conference was unsparing: *"AI large models have caused three major changes in data: from large to small, from cold to hot, from passive to active. Carrying out systematic defense is the only way out."*

He proposed a four-layer defense architecture:

1. In-depth defense for AI application scenarios

2. Full-chain protection for data flow scenarios

3. Dynamic strategy for AI content generation scenarios

4. Three-level linkage for security event response

But as of May 2025, deployment vastly outpaced protection.

The Paradox: More AI Literacy = Less Hallucination Awareness

Perhaps the most counterintuitive finding from the Shanghai Jiao Tong research is the relationship between AI literacy and hallucination detection. One might expect that users who understand how large language models work would be better at spotting their errors. The data shows the opposite.

The Confidence Trap

*Table 8: The confidence-illusion gap. Users who rate themselves as AI-literate are significantly worse at detecting hallucinations than beginners. "Gap" shows deviation from a naive linear expectation. Source: Shanghai Jiao Tong University wave 3.*

The mechanism, as the researchers explain, is confidence inflation. Users who have learned to write prompts, adjust temperature settings, and iterate on outputs develop a sense of mastery that extends to model reliability. They confuse "I know how to use this tool" with "I know when this tool is wrong."

This is particularly dangerous for the demographic that China's AI companies most covet: educated, urban, high-frequency users—the very people most likely to share AI-generated content on social media, use it for work decisions, and recommend it to others.

The Generational Divide

The study also revealed a stark age gradient:

*Table 9: The generational AI literacy gap. Older users face both low literacy AND declining confidence in technology controllability, creating a vulnerability spiral.*

For elderly users—the group that also represents 38% of deepfake fraud victims—the combination of low AI understanding and high life savings creates a perfect target profile. The "dual low" phenomenon (low literacy + low controllability confidence) paradoxically doesn't make them more cautious; instead, it produces technology avoidance anxiety that can be exploited by fraudsters posing as "helpful" technical support.

What Tech Giants Are (Not) Doing About It

China's AI companies have built extraordinary products. What they haven't built—at least not visibly—is comprehensive user protection against the risks their products create.

The Platform Response Gap

*Table 10: Current safety features across major Chinese AI platforms (May 2025). Most hallucination warnings are buried in terms of service rather than integrated into the user experience.*

The pattern is consistent: Chinese AI companies have prioritized capability over safety UX. Hallucination warnings, when they exist, are typically:

- Buried in terms of service documents

- Displayed once during onboarding and never again

- Generic ("AI may produce inaccurate information") rather than contextual

- Absent from high-stakes use cases (medical, legal, financial queries)

Compare this to the approach emerging in some Western platforms: real-time confidence scoring, source citation with verification links, and explicit "this appears to be speculative" labels on uncertain outputs.

The Regulatory Context

China has not been idle on AI governance. The Regulations on the Management of Deep Synthesis Internet Information Services (effective January 2023) and the Generative AI Management Measures (August 2023) established frameworks for content labeling and provider accountability.

But enforcement has been uneven:

- Content labeling requirements for AI-generated media exist on paper but are inconsistently applied

- Platform liability is established but penalties for non-compliance remain modest

- User education mandates are absent from current regulations

- Technical standards for deepfake detection are still in development

The AI-Generated Content Labeling Management Rules, drafted for release by late 2025, would require irremovable watermarks on deepfake audio and video. But by the time they take effect, the technology may have evolved beyond current detection methods.

The Global Lesson: Trust Without Verification

China's hallucination blindness crisis isn't unique to China. It represents an early-warning signal for every nation deploying generative AI at scale. But China's particular circumstances—speed of adoption, scale of deployment, and concentration of government use—make it the most important case study.

Why China Is the Canary

*Table 11: China's position as the global test case for large-scale AI hallucination and fraud risks.*

The global implications are threefold:

First, if 249 million users in a relatively tech-literate population can't detect AI hallucinations, what happens when the next billion come online in India, Southeast Asia, Africa, and Latin America?

Second, the deepfake fraud playbook being perfected in China—exploiting social trust, using video calls, targeting the elderly—is directly exportable. Criminal networks don't respect borders.

Third, government AI deployment without adequate security creates systemic vulnerabilities that could be exploited by state and non-state actors for disinformation, espionage, or sabotage.

A Path Forward: From Blind Trust to Informed Use

The Shanghai Jiao Tong researchers and Qianxin security architects agree on at least one thing: the current trajectory is unsustainable. But their proposed solutions diverge in important ways.

The Education Imperative

Professor Xu Jian's prescription is structural: embed AI literacy into public education at middle school and university levels. Not tool training—critical thinking:

*Table 12: A proposed multi-layer AI literacy architecture combining education and product design.*

The 2023–2025 trend offers a sliver of optimism: the percentage of respondents who believed they "could evaluate AI capabilities and limitations" rose from 59.7% to 85.7%. People are learning—but the gap between self-assessed capability and actual detection skill remains vast.

The Technical Fix

Qianxin's security framework—"接入放心用" (Access with Confidence)—emphasizes technical controls over user education:

1. Red zone isolation: Physically and logically separate AI systems from sensitive networks

2. Identity perimeter: Zero-trust architecture with strict role-based access

3. Application guardrails: Real-time input/output filtering, prompt injection detection

4. Content audit: Automated and human review of AI-generated outputs

5. Incident response: Three-tier escalation for detected anomalies

For government deployments, this means AI models should run on hardened, air-gapped infrastructure with human-in-the-loop verification for consequential decisions—not the current pattern of rapid deployment with security as an afterthought.

The Hard Truth

Neither education nor technical controls will fully solve the hallucination problem. Large language models will continue to produce confident-sounding falsehoods. Deepfake technology will continue to improve. The arms race between detection and generation has no foreseeable endpoint.

What can change is the social contract around AI use:

- Users who understand that AI is a tool, not an oracle

- Platforms that design for skepticism, not blind trust

- Regulators who treat AI safety as infrastructure, not optional compliance

- A culture where "AI said so" is never sufficient justification for consequential decisions

What the Internet Is Saying

"我用Kimi写论文大纲，它给我编了三个不存在的参考文献。幸好我查了一下，不然答辩就完蛋了。"

*"I used Kimi to write my thesis outline. It fabricated three references that don't exist. Luckily I checked, or my defense would have been a disaster."*

— Xiaohongshu user, thesis writing thread

"我妈接到'我'的视频电话说要交学费，差点转了5万。现在的AI太可怕了。"

*"My mom got a video call from 'me' saying I needed tuition money. She almost transferred ¥50,000. AI is terrifying now."*

— Weibo user, AI fraud awareness post

"真正的危险不是AI会撒谎，而是人类已经懒得去验证了。"

*"The real danger isn't that AI lies—it's that humans have become too lazy to verify."*

— Zhihu comment, AI ethics discussion

"奇安信说90%的DeepSeek服务器裸奔，这让我想起当年摄像头默认密码是admin的日子。历史总在重复。"

*"Qianxin says 90% of DeepSeek servers are naked. Reminds me of the days when security cameras all had default password 'admin.' History repeats."*

— V2EX developer forum

"学历越高的人越容易被AI骗？这研究太扎心了。读了二十年书，结果被一个模型忽悠了。"

*"Higher-educated people are MORE likely to be fooled by AI? This study hits hard. Twenty years of schooling, then get conned by a model."*

— Douban discussion, tech reading group

"政府急着部署AI，安全却跟不上。这不是创新，这是在赌国运。"

*"Governments are rushing to deploy AI while security can't keep up. This isn't innovation—it's gambling with national fate."*

— WeChat public account, cybersecurity commentary

Related Articles

- China's AI Deepfake Fraud Crisis: How 700,000 Annual Scams and a $40 Billion Global Threat Are Reshaping Trust

- DeepSeek's $7.3 Billion Megaround: Inside China's AI Funding Frenzy

- China's AI War of Gods: DeepSeek, Kimi, and the Open-Source Path to Dominance

- The End of Free AI: How Doubao Charging Marks a New Era in China's Token Economy

- Stanford AI Index 2026: The Statistical Case for China's Rise

*Published May 17, 2026. Data current as of the 2025 Data Security Development Conference and Shanghai Jiao Tong University Science Communication Conference. For corrections or data updates, contact the editorial team.*