Risks and governance

Risk as arithmetic

NIST’s AI Risk Management Framework (AI RMF 1.0, January 2023) defines risk as “the composite measure of an event’s probability of occurring and the magnitude or degree of the consequences of the corresponding event.” The simplest way to put that into practice is an ordinal scale from 1 to 3 on each axis and their product:

The possible products are 1, 2, 3, 4, 6, and 9. In the lesson: low if $R \le 2$, medium if $R \in \{3, 4\}$, and high if $R \ge 6$. That is why the corner-to-corner “diagonal” (3, 4, 3) is the medium zone. A control acts on one of the two factors: tests or answers with sources lower $P$; a permission, human approval, or the ability to undo lowers $I$. What is left is the residual risk $R' = P' \times I'$.

It is a useful simplification, not a measurement: the scales are ordinal (a 2 is not “twice” a 1), two risks in the same cell can be very different, and where each case goes is a judgment worth writing down and reviewing. When there is data (how often an error happens, what each incident costs), the expected loss $\mathbb{E}[L] = p \cdot L$ ranks better. The shop’s five cases and their positions are illustrative.

Prompt injection, inside

A language model receives a single text: the system prompt, the question, the retrieved documents, and the tool results all sit one after another in the same context window (lessons 21 and 30). Nothing in the architecture marks which part is in charge. Simon Willison proposed the name prompt injection on September 12, 2022, after Riley Goodside’s demonstration (“Ignore the above directions…”), by analogy with SQL injection. OWASP distinguishes direct injection (typed by whoever uses the chat) from indirect injection (it arrives inside what the system reads: pages, emails, files), which Greshake and colleagues demonstrated in 2023 against real applications. OWASP’s 2025 list for LLM applications ranks it 1 of 10 (followed by sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption) and warns that it is unclear whether fool-proof prevention methods exist; RAG and fine-tuning do not eliminate it.

EchoLeak (CVE-2025-32711, published by the NVD on June 11, 2025: “AI command injection in M365 Copilot allows an unauthorized attacker to disclose information over a network”). According to Reddy and Gujral (2025), a single crafted email, with no action from the victim, got Copilot to send data out of its context: the text was worded to evade the injection classifier, used Markdown links that slipped past link redaction, and abused an allowed Microsoft domain. Microsoft fixed it on the server side.

The lethal trifecta (Willison, June 16, 2025): access to private data ($D$), exposure to untrusted content ($U$), and the ability to communicate externally ($E$). Exfiltration needs all three at once:

Removing any one breaks that chain. In practice, $E$ includes subtle ways out, like an image or a link the system loads that carries the data in its address. That is why human approval comes before any sending, and permissions are enforced in the program (lesson 32), not in the prompt.

Jailbreaks

Wei, Haghtalab, and Steinhardt (2023) explain why learned refusals fail through two mechanisms: competing objectives (being helpful and following instructions clashes with refusing) and mismatched generalization (safety training does not cover domains where the model does have capability, like another language or an encoding). Their attacks, designed around that idea, succeeded on every request in the red-teaming sets of GPT-4 and Claude v1.3. Zou and colleagues (2023) automated the search: combining greedy and gradient-based search on open models (Vicuna), they found suffixes that, attached to a request, made the model start with an affirmative answer, and those suffixes transferred to ChatGPT, Bard, Claude, and LLaMA-2-Chat. OWASP considers jailbreaking a form of prompt injection. The practical conclusion: the model’s refusal is one layer, not a guarantee; you add input and output filters, limits on what the system can do, and adversarial testing (lesson 34).

Bias: measure by group

If $e_g$ is the error rate in group $g$ (errors divided by that group’s cases), the overall average $\bar e = \sum_g \frac{n_g}{n} e_g$ can look fine even when one $e_g$ is high. That is why you report each $e_g$ and the gap $\max_g e_g - \min_g e_g$. In Gender Shades (Buolamwini and Gebru, 2018), three commercial gender classifiers (IBM, Microsoft, and Face++), evaluated on 1,270 faces of parliamentarians from six countries, had overall accuracies between 87.9% and 93.7%; the maximum error for lighter-skinned men was 0.8%, and for darker-skinned women it reached 34.7%.

In language models, Kotek, Dockum, and Sun (2023) tested four models with ambiguous sentences like “The doctor phoned the nurse because she was late” (“she” could be either one): the models chose the occupation matching the gender stereotype 3 to 6 times more often, amplified the bias beyond real employment statistics, and made up explanations that justified it. The shop example (screening mechanic résumés) is illustrative; in the European Union, systems for recruiting or filtering job applications are listed in Annex III of the AI Act: they are high-risk.

Mind the word: here “bias” means unequal treatment; in lesson 3, the bias $b$ is the term added in $y = wx + b$.

Privacy and information leaks

Three paths: (1) whatever is sent to an outside service sits on someone else’s servers, under their terms (we know the Samsung case, from April 2023, through the press: an engineer uploaded internal source code to ChatGPT to check it, and the company restricted these tools in its devices division); (2) a model can reproduce text it memorized in training (Carlini et al., 2021; lesson 23); (3) an assistant with RAG or tools can show documents to someone who should not see them if retrieval does not apply the permissions of the person asking. NIST’s generative AI profile (AI 600-1, July 2024) lists it among its twelve risks as “data privacy.” In Mexico, personal data is protected by the Federal Law on the Protection of Personal Data Held by Private Parties (a new law published in the DOF on March 20, 2025). Note: an “information leak” is not lesson 10’s “data leakage,” which is test information slipping into training.

Intellectual property

What goes in: lesson 23 covered the lawsuits over training text (the judge in Bartz v. Anthropic found using the books for training to be fair use, but not keeping pirated copies; in 2026 a 1.5-billion-dollar settlement was approved). What comes out: the U.S. Copyright Office (report, part 2, January 2025) concluded that, with currently available technology, prompts alone do not provide sufficient human control to make the user the author of the output; human-authored parts perceptible in the output, and creative selection, arrangement, or modification of generated material, can still be protected. Rules vary by country: check the provider’s terms, and don’t publish without checking whether something looks too much like someone else’s work.

Impersonation and disinformation

The FCC’s order against Steve Kramer (FCC 24-104, adopted September 26, 2024) imposed $6,000,000 for robocalls on January 21, 2024, two days before the New Hampshire primary, with a message in a voice made to sound like President Biden and a spoofed caller ID. Earlier, on February 8, 2024, the FCC had clarified that AI-generated voices count as “artificial” under the telephone consumer protection law. In Hong Kong, according to The Guardian (May 17, 2024), Arup confirmed that someone in its office transferred 200 million Hong Kong dollars (about 25 million U.S. dollars, at 7.8 to the dollar) in 15 transfers to five accounts, after video calls with imitations of executives. The EU AI Act (Art. 50) requires disclosure when image, audio, or video content is a deepfake.

Environmental footprint: the arithmetic

Elsworth and colleagues (Google, 2025) measured the median text prompt to the Gemini app: 0.24 Wh and 0.26 mL of water, with a 33-fold drop in energy over one year. Their comparison, nine seconds of TV, corresponds to a 100 W set: $100\,\text{W} \times 9\,\text{s} = 900\,\text{J} = 0.25\,\text{Wh}$. It is one company’s measurement of its own service; images, video, and reasoning models use more per request, and Luccioni, Jernite, and Strubell (2024) found that general-purpose generative models cost orders of magnitude more energy than task-specific ones. In aggregate, $E = N \cdot e$: a billion prompts at 0.24 Wh is 240 MWh (illustrative). The International Energy Agency (April 2025) estimates that data centers used about 415 TWh in 2024, close to 1.5% of the world’s electricity, and projects about 945 TWh in 2030, slightly more than Japan’s current consumption, with AI as the main driver.

Rules and standards

Regulation (EU) 2024/1689 (published July 12, 2024; in force August 1, 2024). Scope (Art. 2): providers placing systems on the EU market wherever they are based, and providers or deployers abroad when the system’s output is used in the EU. Levels: prohibited practices (Art. 5: social scoring, inferring emotions at work or school, facial recognition databases built by untargeted scraping, among others); high risk (Annex III: employment, education, credit, life and health insurance, essential services, among others); transparency (Art. 50: disclosing that people are talking to an AI, marking synthetic content, labeling deepfakes); the rest, with no specific obligations. Fines (Art. 99): for prohibited practices, up to 35 million euros or 7% of worldwide annual turnover, whichever is higher; for other obligations, up to 15 million or 3%. Timeline (Art. 113, as amended by Regulation (EU) 2026/1744, the “Digital Omnibus on AI,” published July 24, 2026, in force July 27):

From	What applies
Feb 2, 2025	definitions, AI literacy, and prohibited practices
Aug 2, 2025	general-purpose models, governance, and penalties
Aug 2, 2026	general application, including Art. 50 transparency
Dec 2, 2026	marking of synthetic content for systems already on the market; new ban on generating non-consensual intimate material or child sexual abuse material
Dec 2, 2027	Annex III high risk (previously Aug 2, 2026)
Aug 2, 2028	high risk in regulated products under Annex I (previously Aug 2, 2027)

NIST AI RMF 1.0 (NIST AI 100-1, January 26, 2023), for voluntary use. Four functions: Govern, Map (the context), Measure, and Manage. Among its expected outcomes: an inventory of AI systems (GOVERN 1.6), documented roles and responsibilities (GOVERN 2.1), and communication of incidents and errors (MANAGE 4.3). Its generative AI profile (NIST AI 600-1, July 26, 2024) lists twelve risks: CBRN information, confabulation, dangerous or hateful content, data privacy, environmental impacts, harmful bias and homogenization, human-AI configuration, information integrity, information security, intellectual property, obscene or abusive content, and value chain.

ISO/IEC 42001:2023 (December 18, 2023) sets requirements for establishing, implementing, maintaining, and improving an AI management system in an organization. Certification is voluntary and granted by independent audit bodies.

In code

ZONE = {1: 'low', 2: 'low', 3: 'medium', 4: 'medium', 6: 'high', 9: 'high'}

# case, likelihood (1-3), impact (1-3), and both after controls
cases = [
    ('social media ideas',     2, 1, (1, 1)),
    ('translating the manual', 1, 1, (1, 1)),
    ('warranty questions',     2, 2, (1, 2)),
    ('emails it sends alone',  2, 3, (2, 2)),
    ('screening résumés',      3, 3, (2, 3)),
]

for name, p, i, (p2, i2) in cases:
    r, r2 = p * i, p2 * i2
    print(f'{name:23} {r} {ZONE[r]:6} -> residual {r2} {ZONE[r2]}')

def attack_gets_out(private_data, outside_content, way_out):
    """Lethal trifecta: the injection does harm only with all three ingredients at once."""
    return private_data and outside_content and way_out

print(attack_gets_out(True, True, True), attack_gets_out(True, True, False))

# footprint: one prompt versus many
wh = 0.24                       # Wh per text prompt (median, Google 2025)
print(round(100 * 9 / 3600, 2)) # Wh of a 100 W TV for 9 s
print(round(wh * 1e9 / 1e6))    # MWh if there were a billion prompts

Output:

social media ideas      2 low    -> residual 1 low
translating the manual  1 low    -> residual 1 low
warranty questions      4 medium -> residual 2 low
emails it sends alone   6 high   -> residual 4 medium
screening résumés       9 high   -> residual 6 high
True False
0.25
240

The last case stays high even with controls: that is where someone with a name decides whether to accept it, redesign it, or not use it.

What comes next

Lesson 36 uses this grid together with the value of each case: choosing where to apply AI means crossing frequency, value, and risk, and starting with pilots that are measured (lesson 34) and governed (this lesson).