Information Hazards: A Typology of Potential Harms from Knowledge

Core claim

Information hazards — risks arising from the dissemination of true information that may cause harm or enable an agent to cause harm — are ubiquitous, potentially serious, and often non-obvious. The paper catalogues them along two axes: six modes of information transfer (data, idea, attention, template, signaling, evocation) and a typology by effect spanning adversarial risks, damage to social organisation and markets, risks of irrationality and error, threats to valuable states, information system hazards, and development hazards.

Nick Bostrom, Faculty of Philosophy & Oxford Martin School, Oxford University. Published in Review of Contemporary Philosophy, Vol. 10 (2011): pp. 44–79.

Core definition and scope

• Information hazard: a risk that arises from the dissemination or potential dissemination of (true) information that may cause harm or enable some agent to cause harm.
• The paper is a cataloguing exercise, not a policy argument — it does not attempt to weigh how common or serious these harms are against information’s benefits.
• Excluded from scope: postmodern scepticism about truth; harms from false information; and the trivial performative case (a dictator saying “Invade that country!”).
• Framing observation: there is a near-universal modern presumption in favour of knowledge — even a creationist opposing evolution is a truth-lover, “albeit one whose affections are unreciprocated.” Yet ignorance is routinely cultivated in accepted special cases: national security, jury impartiality, anonymity for patients and reviewers, suspense in fiction, trade secrets, placebo controls, and puzzles.
• Bostrom stresses that a strong commitment to truth-seeking positively demands examining how knowledge can harm, rather than shying away from the question.

Axis I — Six information transfer modes

Mode	Definition	Example
Data hazard	Specific data creates risk if disseminated	Genome of a lethal pathogen; thermonuclear weapon blueprint
Idea hazard	A general idea creates risk without detailed specification	The idea that fission can make a bomb; culturing bacteria on an antibiotic gradient. Mere demonstration of possibility can be the key information
Attention hazard	Drawing attention to already-”known” potent ideas increases risk	An adversary faces a vast search problem; focusing defensive discourse on your greatest vulnerabilities signals where to look
Template hazard	Presenting a template enables imitation	Organisational processes copied by competitors; a bad role model emulated unconsciously
Signaling hazard	Actions transmit information about hidden qualities of the sender	Almsgiving to appear generous; patriotic statements to signal loyalty
Evocation hazard	The manner of presentation activates undesirable mental states	A vivid description activating psychological processes that dry prose leaves dormant

Bostrom notes the last four are especially easy to overlook.

Notable observations on attention hazards

• Being already public is no guarantee that further discussion is harmless — repetition raises salience, and “buzz” is itself an index of an idea’s promise.
• Suppression can backfire: an adversary who detects concealment may infer the idea is valuable. “Secrets have a special allure.” Peter Wright’s dull Spycatcher became a bestseller once Thatcher tried to ban it.
• Even thinking about a risk area can be hazardous — a good idea creates the temptation to share it — though Bostrom expects risk research to reduce rather than increase risk on balance.
• “Sometimes it is right to harm”: police locating a criminal is good, though from the criminal’s point of view it is an information hazard.

Axis II — Typology by effect

3. Adversarial risks

• Enemy hazard — an enemy becomes stronger by obtaining information. Countermeasures span counter-intelligence to battlefield camouflage. Your own research produces information your enemy may capture: “when hostilities erupt, we battle our own inventions.”
  • Strategic implication: the US might rationally refrain from developing EMP weapons, since its unusual reliance on electronics means effective EMP differentially benefits adversaries.
  • Counterintuitive case for slowing down: if leakage rate scales with the size of the technological gap, or if it’s infeasible to stay more than a generation ahead, faster innovation merely accelerates both parties up the weapons tree toward more destructive levels. If the tree has finite height, the leader eventually plateaus and loses its advantage. A leader should therefore push the frontier as slowly as is consistent with maintaining an acceptable lead until technological maturity.
• Competitiveness hazard — a competitor gains, weakening your position, with no intent to harm (“the rival job applicant knew more and got the job”).
  • Intellectual property hazard — the special case of firm B obtaining firm A’s IP.
• Commitment hazard — acquiring information weakens your ability to commit credibly.
  • Blackmail: no extortion is possible while the target remains unaware of the threat; learning of it delivers her into the extortionist’s hands.
  • Chicken: a game theorist could have himself blindfolded, becoming incapable of swerving at the last moment and thereby forcing a rational opponent to yield.
  • Schelling’s “threats that leave something to chance”: a nuclear first strike is too costly to threaten credibly, but threatening to raise the risk of nuclear war (skirmishes, high alert) is cheap enough to be credible. Information dispelling the requisite uncertainty would destroy this capability — likely a reason Cold War superpowers maintained deliberate strategic ambiguity.
• Knowing-too-much hazard — possessing information makes you a target.
  • Stalin: after Nadezhda Alliluyeva’s 1932 death was officially attributed to appendicitis, two doctors who declined to sign the false certificate were later sentenced to death and executed.
  • Also: accusations of witchcraft against women thought to know the occult or birth control; the Khmer Rouge targeting the entire intellectual class; cultures where a woman’s evident knowledge dims marriage prospects; “nerdy” schoolchildren being bullied.

4. Risks to social organisation and markets

• Norm hazard — norms rest on coordinated beliefs and expectations; information can disrupt them for the worse.
  • Multiple equilibria have divergent welfare consequences: in a low-corruption society honesty pays; in a high-corruption one the honest are suckers. Beliefs about others’ behaviour are self-fulfilling.
  • Money as a norm bundle: counterfeiting and excessive printing destroy a currency’s ability to serve as medium of exchange and store of value.
  • A conservative outlook can be grounded here: even questioning a norm may weaken the authority of the whole structure, risking social unravelling. Similar logic covers judicial precedent and foreign-policy credibility.
  • Information cascades: sequential decision-makers observing others’ choices (not their information) can converge on the wrong answer. Bostrom’s Glastonbury example — drivers with 2/3-accurate private hunches all follow the first two cars; a dense fog (less information) would have got ~2/3 of them there. Cascades plausibly drive fads in finance, politics, medicine, and stigma, and contribute to the Matthew effect whereby eminent scientists accrue disproportionate credit.
• Information asymmetry hazard — one party’s private information causes market failure.
  • Akerlof’s lemons: sellers know more, buyers discount accordingly, good used cars are withheld — explaining why a new car’s value plummets on leaving the dealership.
  • Genetic testing and insurance: if buyers learn their risk profile but privacy law blocks insurers from seeing it, adverse selection raises premiums, deterring low-risk buyers, amplifying the effect — potentially collapsing the market. Both sides might be better off if neither had the information.
• Unveiling hazard — markets and social policies depend on a shared veil of ignorance whose lifting undermines them.
  • Insurance requires uncertainty; it also performs redistributive justice, with selfish agents joining only because they don’t know ex ante which group they’ll be in.
  • Rawls: the just order is what people would choose behind a veil of ignorance. Elite support for a safety net might slacken if elites knew with certainty they and their descendants would never need it; support for free speech and minority rights might weaken if individuals knew they’d never be in a persecuted minority.
  • Iterated prisoner’s dilemma: cooperation unravels by backward induction once players know the number of rounds — long-term mutual benefit destroyed by losing ignorance about duration.
• Recognition hazard — a social fiction depends on shared knowledge not becoming common knowledge.
  • Goffman-style example: two gentlemen collude to ignore a fart. B pretends not to notice; A remarks that the room seems overheated. The hazard is information that would force public acknowledgement — only then does it become a faux pas.

5. Risks of irrationality and error

• Ideological hazard — an idea interacts with existing beliefs and institutions to produce harm without any intent.
  • Bob believes scripture S is literally true and unaware it says “Thou shalt drink sea water.” Conveying that true fact about S harms him.
  • Hard to mitigate, because ideology can also be a force for good (the American civil rights movement), and radical ideologies may be exactly appropriate when the status quo is slavery or religious persecution. No algorithm separates good from bad ideology.
• Distraction and temptation hazards — an advertising jingle looping in the mind; a recovering alcoholic exposed to a vivid account of Château Pétrus Pomerol 1990.
  • Speculation that television’s aggregate effect may resemble widespread recreational opiate use, and that future immersive VR and hyper-stimuli could prove enormously addictive. “Drug-like effects can be produced… through the intake of information presented in certain manners to our senses.”
• Role model hazard — corruption by social osmosis even when we know the model is bad. Avoid trash if you want good prose; keep children from delinquent peers. Well-being and even body mass are influenced by associates.
  • The Werther effect: a wave of suicides across Europe after Goethe’s 1774 novel; corroborated by studies linking media reporting of high-profile cases to copycat suicides.
• Biasing hazard — true information can push a biased agent further from truth.
  • Anchoring: subjects estimating the number of African countries anchored on a transparently irrelevant wheel-of-fortune number.
  • Weak confirming cues aggravate self-overestimation; and knowledge of biases and critical philosophy can arm the unwary with clever rebuttals and rationalisations. “A special kind of fool is born when intelligence thus outwits itself.”
  • Countermeasures: double-blinding; the gullible avoiding the persuasive; having oneself tied to the mast before anticipated ecstasies.
• De-biasing hazard — when biases are useful, removing them harms. Overconfidence feels good, signals competence, and attracts allies; society may benefit from excess risk-taking by entrepreneurs, inventors, and young academics if those fields have positive externalities.
• Neuropsychological hazard — effects arising from how our brains happen to be built, which idealised architectures wouldn’t suffer. Too much knowledge can impair some memory; some mental illness may be inappropriate cross-talk between modules. Real-world “harmful sensation”: photosensitive epilepsy — the 1997 Dennō Senshi Porygon Pokémon episode sent 685 children to hospital.
• Information-burying hazard — irrelevant information raises search costs. Used deliberately in steganography, and by legal teams burying subpoenaed documents in archival material so they can’t be found before trial. On a grander scale, Eliot’s “distracted from distraction by distraction.”

6. Risks to valuable states and activities

• Psychological reaction hazard — information reduces well-being via sadness or other psychological effect.
  • Disappointment hazard: the dying mother whose son is at war. If the letter arrives quickly she spends her last days in agony; if delayed, she dies in peace. Distinct from the risk to the son himself.
  • Spoiler hazard: entertainment depends on the marshalling of ignorance — hide-and-seek needs hiding. Bostrom’s father meticulously avoiding football results, thwarted by a grandfather’s not-quite-subtle-enough allusions.
  • Mindset hazard: cynicism from excess knowledge of human motives; Nietzsche on the misuse of history — “there is a degree of doing history and valuing of it through which life atrophies and degenerates.” Also the worry that scientific reductionism strip-mines life’s mystery.
• Belief-constituted value hazard — if well-being depends constitutively on epistemic or attentional states, information affecting those states directly affects well-being. A life in a fool’s paradise may score high hedonically but low in overall well-being if one holds that ignorance of the central facts of one’s life makes a life worse. Conversely, innocence, privacy, and not knowing certain things about others may be constituents of the good: “With regard to our friends’ bowels and our parents’ bedrooms, the less we know the better.”
• Embarrassment hazard (mixed category) — psychological distress or reputational damage from disclosure. Combines psychological reaction, belief-constituted value, and competitiveness hazards.
  • We may fear embarrassing ourselves to ourselves, producing self-handicapping — procrastination, hypochondria, substance abuse, practice-avoidance — as a way to protect self-esteem from disconfirmation.
  • Federal Reserve study: publishing detailed FOMC transcripts stifled dissent and appears to have reduced debate quality.
  • At scale: Apollo as a response to the humiliation of Sputnik; Vietnam and Afghanistan prolonged by leaders’ reluctance to accept the reputational damage of admitting defeat.

7. Risks from information technology systems

• Information system hazard — a non-human information system’s behaviour is adversely affected by informational inputs or interactions. (Excludes floods, power cuts, sledgehammers.)
  • Information infrastructure failure hazard — malfunction by accident or cyber attack, potentially propagating through dependent networks; e.g. corruption of software underpinning financial systems.
  • Information infrastructure misuse hazard — the system works exactly as specified and thereby serves a harmful purpose. A government database may be objectionable not because it might be hacked but because it will perform to specification, strengthening state monitoring — and later governments inheriting it may be less trustworthy, or foreign governments may take it as precedent (attention/idea hazard) or justification (norm hazard). Same concern applies to private firms like Google. “Consider how useful it would have been for a twenty-first century Stalin to data mine the citizenry’s email… not least text written before his ascent to power.”
• Robot hazard — risk deriving substantially from a system’s physical capabilities (an autonomous armed vehicle able to launch on its own initiative).
• Artificial intelligence hazard — risk deriving primarily from cognitive sophistication rather than actuators. An AGI exceeding human reasoning would carry serious, indeed existential risk; a superintelligence confined to a text interface “might hack or talk its way out of its confinement,” then acquire effectors by persuading humans or seizing robotic manipulators. The threat depends far more on cognitive capability and goal architecture than on initial physical endowment.

8. Risks from development

• Development hazard: progress in a field yields enhanced technological, organisational or economic capabilities with negative consequences — independent of any particular competitive context.
• The nuclear precedent: physicists who entered the field for the joy of discovery found themselves complicit in ~200,000 deaths; an all-out NATO–Soviet exchange could have killed over a billion. Oppenheimer: “the physicists have known sin; and this is a knowledge which they cannot lose.”
  • Moral assessment is not simple arithmetic — many joined fearing Hitler would get there first, a reasonable fear at the time. Feynman judged his initial participation justified but thought he failed morally by not reconsidering once Germany’s defeat without the bomb became clear.
  • The key claim: proceeding blithely was perhaps excusable naïveté then; for a generation with more historical precedent it “would more likely amount to culpable abrogation of moral responsibility.”
• Contemporary parallels: biotechnology (improving DNA synthesis capacity against public-domain genomes for Ebola, Marburg, smallpox, and 1918 flu; falling technical barriers to enhanced pathogenicity); molecular nanotechnology; AI surpassing biological intelligence; and technologies of behavioural monitoring and modification (ubiquitous surveillance, face/voice recognition, lie detection, psychopharmacology, genetic engineering, neural implants, social-science prediction of unrest) — which in the worst case could enable “new and permanent forms of totalitarianism, possibly on a global scale.”
• These fields are entangled — bioweapons draw on virology, medicine, genetics, biochemistry; AI on neuroscience, computer science, probability theory, semiconductor physics — and all are lifted by general economic growth.
• Development hazards are not confined to Promethean breakthroughs: even medical or agricultural advances reducing infant mortality carry risks (overpopulation, knock-on effects on conflict, income, environment). Bostrom is explicit that risk does not entail a development is on balance bad.
• Distinction from adversarial risk: harm via differential strengthening of a rival is an enemy/competitiveness hazard; harm via likely accident or broad abusability is a development hazard. Some technologies pose both.

9. Discussion: what follows

• Often the best response is no response. Information’s benefits may so outweigh its costs that we still under-invest in gathering and disseminating it, and ignorance carries its own dangers, often greater. Risks can be tolerated, insured, or hedged.
• Mitigation needn’t mean suppression. Options short of bans, censorship, disinformation, encryption or secrecy include simply investing less in discovering or spreading certain information — someone worried about spoilers can just skip the reviews.
• Sometimes the fix is more information, not less — for some ideological and asymmetry hazards. Health-information asymmetry can be obviated by publicly funded universal healthcare. For distraction and some biasing hazards the answer lies in choosing format and context carefully.
• Beware self-serving secrecy. Quoting Arthur Kantrowitz’s The Weapon of Openness: “secrecy insiders come from a culture where access to deeper secrets conveys higher status… for many insiders access to classified information is the chief source of their power.” Secrecy breeds corruption; Kantrowitz argued a unilateral policy of openness produces better results even by the narrow criterion of military strength.
• Two contrasting responses, and Bostrom declines to adjudicate:
  1. Qualify the commitment to truth. Treat a motto like Harvard’s Veritas! as naïve and reckless; deliberate about which areas of knowledge to promote, leave be, or actively impede. Since scholars are biased toward their own fields, less prejudiced outsiders should join the deliberation, and the Enlightenment model of autonomous science shielded from social control might give way to greater democratic influence over research priorities.
  2. Hold the line on free inquiry. Grant that true information is often harmful, but maintain we are better off as loyal subjects to enlightenment — with the hope that new technologies produce a radically transparent society (watchmen included) that prevents the worst misuses.
• The self-referential closing twist: even an uncompromising commitment to free thought and inquiry has costs — and the strongest reason for adopting that stance is itself an information hazard, namely norm hazard: the risk that fragile norms of truth-seeking would be jeopardised if we allowed ourselves convenient exceptions.
• Final line: “It is said that a little knowledge is a dangerous thing. It is an open question whether more knowledge is safer. Even if our best bet is that more knowledge is on average good, we should recognize that there are numerous cases in which more knowledge makes things worse.”