Cybersecurity of AI: Adversarial Attacks and LLM Privacy

1. The ML system as an asset: two attack surfaces

A classifier maps inputs to labels; training selects parameters using an objective and data. Neural networks do not universally outperform every other method. When an application relies on a model, protect the model, data and surrounding workflow as assets. Clean test accuracy alone does not establish security under adversarial conditions.

Training-time attacks and inference-time attacks involve different capabilities. For example, poisoning changes the learning process or its data, while an evasion example changes an input presented for prediction. A threat model must specify the adversary’s access, objective and constraints; a vulnerability and an attempted attack are not the same as a successful system compromise.

Controls address different threats: input robustness, access restrictions, isolation and privacy mechanisms are not interchangeable guarantees. Evaluate the relevant component and then the application, including how it uses model output.

2. Adversarial examples: the vulnerability and the attack

A classifier assigns an input to a label. Accuracy on sampled test data does not establish stability under every allowed perturbation. An attacker’s objective, input access and constraints determine what counts as a successful adversarial example.

An adversarial example is an intentionally constructed inference-time input that causes an unwanted model outcome under a declared threat model. A small, label-preserving perturbation is one common setting. Imperceptibility, transfer to another model and cheap construction are not universal requirements; visible adversarial patches are another setting. See Brown et al., Adversarial Patch.

In Goodfellow et al., Figure 1, one GoogLeNet input changes from panda (57.7% class score) to gibbon (99.3%) after a perturbation with ε = 0.007 in that model’s encoding. These are scores for one illustration, not success rates or a guarantee of invisibility for arbitrary images. See the model-specific FGSM results.

What the neuron experiment does—and does not—show

Szegedy et al., §3 found semantically related activating images for both individual high-level units and random linear combinations in their examined networks. This challenges treating individual coordinates as the uniquely meaningful basis. It does not establish that individual neurons can never be interpretable, or that neural networks learn no human-recognizable features.

A feature can be an input-derived function or a representation direction, not necessarily one neuron. Distributed representation, interpretability and robustness are different questions. The useful/non-robust distinction in section 7 is defined relative to data and allowed perturbations, not by whether a feature looks meaningful to a person.

Random corruption is not worst-case perturbation

A random-noise test averages over a chosen noise distribution; an adversarial robustness claim concerns all allowed changes or an explicitly evaluated attack. Random draws may miss damaging directions, but random noise is not universally harmless. Successful attacks are not necessarily minimal, cheap, imperceptible or guaranteed to transfer.

For a linear score, signed margin and the perturbation budget determine whether a label can change; dimensionality alone is not enough. The exact calculation in section 7 makes this distinction concrete.

From model error to system impact

A manipulated decision can affect integrity; consequences for availability, confidentiality or physical safety depend on how the surrounding application uses that decision and on its other controls. A benchmark misclassification alone does not demonstrate a system intrusion or an accident. Clean accuracy and robustness should therefore be evaluated separately under the intended deployment conditions.

3. Threat models: knowledge, objectives and constraints

A useful threat model states the asset and learning stage, what an adversary knows, what they can change or observe, and how success is measured. The classification examples below concern inference-time evasion against a fixed model; poisoning the training process is a different capability.

Knowledge is not an identity or permission

Terminology depends on scope: NIST AI 100-2e2025, §§2.1.4 and 2.2.1, describes broad full-system knowledge including training data, while its white-box evasion discussion specifies architecture and parameters. State which convention applies. “Insider” describes a trust relationship, not a knowledge level: public weights can expose a model to an outsider, and an insider need not know every parameter. Knowledge does not grant permission to test a service.

How candidates are generated

These are not three mutually exclusive attacker identities. A workflow can query the target to label data, fit a surrogate, then attempt transfer: Papernot et al., §4, uses target labels in this way. Conversely, Brendel et al., §§2–3, describes decision-based search without a substitute. See also NIST, §§2.2.2–2.2.3. “Direct API query” alone does not distinguish score-based from label-only feedback.

Objective is not a severity rating

For this example, fix a classifier f, a clean input x with f(x) = y, and a valid, label-preserving candidate x′. Choose the target t ≠ y before evaluating the candidate.

• Untargeted success: f(x′) ≠ y — any incorrect class meets the objective.
• Targeted success: f(x′) = t — only the selected incorrect class meets the objective.

Under the same assumptions and candidate constraints, every targeted success is also an untargeted success; the converse need not hold. A stricter success condition does not prove that every algorithm takes longer, or that every targeted error causes more damage. In binary classification there is only one incorrect class, so the two conditions coincide.

Make the experiment reproducible

1. Allowed changes: state input representation, valid bounds, perturbation norm and budget where applicable; explain why the reference label is preserved.
2. Available resources: state model/preprocessing knowledge, auxiliary data, query feedback, query limit, restarts and computation budget.
3. Success measure: declare the objective and eligible examples, including treatment of clean errors; report the denominator. For stochastic models, specify repeated evaluation and the success-probability criterion.

Additional capabilities can enable more methods; they do not guarantee that a finite attack run succeeds. Failure to find a candidate is not a robustness proof. The FGSM example and PGD discussion keep optimization steps separate from measured success.

4. FGSM: one input-gradient step

The Fast Gradient Sign Method (FGSM) constructs a candidate using one gradient evaluation. Its linearization argument motivates a fast white-box attack; it does not require a globally linear network or guarantee success.

1. Choose the input representation, its valid bounds, ε and the reference label y.
2. Compute the gradient of J with respect to x, keeping θ and y fixed.
3. Form δ from the component-wise signs.
4. Clip x + δ to the valid input bounds, if the domain is a coordinate box containing x.
5. Evaluate the candidate against the declared attack objective. Increasing loss is not itself success.

For a labeled benchmark, use the ground-truth label and state whether clean errors are excluded from the denominator. The explorer below instead fixes y to its clean prediction, which is explicitly a different, illustrative protocol. Clipping bounds must match the coordinates: [0,1] is appropriate for normalized pixels, not automatically for standardized inputs.

The FGSM algorithm

g = ∇x J(θ, x, y)
δ = ε · sign(g)
x_candidate = clip(x + δ, lower, upper)
Evaluate the candidate; success is not guaranteed.

Why it works. The gradient describes local loss sensitivity; its sign maximizes the first-order approximation over an L∞ box. Input clipping preserves the allowed domain, not perceptual invisibility. Evaluate the candidate: neither a global maximum nor a changed prediction is guaranteed.

Published FGSM results: keep the model and units

Transcribed from Goodfellow et al., ICLR 2015, §4, p. 3 and footnotes 1–2; these experiments were not rerun here.

The scores are the paper’s reported averages, not measured calibration or success rates. Epsilon values in different representations are not directly comparable.

Figure 1 is one GoogLeNet/ImageNet illustration: panda (57.7% class score) → gibbon (99.3%), ε = 0.007 in GoogLeNet input encoding. It supplies no aggregate ImageNet attack-success rate.

Download the transcribed records and source version. The coordinate plot below is a separate editorial example.

FGSM against a policy: observation, action and return

One FGSM step uses a gradient of a chosen loss with respect to the input, followed by a constrained input update. Its cost depends on the model and access available; “one gradient” does not mean universally cheap or a one-gradient black-box attack. A surrogate policy may first have to be trained. The policy parameters stay fixed during the input attack.

Huang et al. (2017), §§4–5 tested trained DQN, TRPO and A3C policies on four Atari games. They perturbed observations at inference time and measured trajectory returns. Their attack loss opposes the clean policy’s highest-weighted action, which is a reference choice—not an oracle for the truly optimal action. For DQN they differentiated a softmax over Q-values while execution still chose the maximal-Q action.

o_t = observe(s_t)
a_ref = argmax_a π(a | o_t)             # freeze this reference
J(o) = -log π(a_ref | o)
o_seen = clip(o_t + ε sign(gradient_o J), input bounds)
a_t = select_action(policy, o_seen)
s_(t+1), r_(t+1) = environment_step(s_t, a_t)
G = sum_t γ^t r_(t+1)

The attack changes what the agent sees, not the current environment state or reward directly. A changed action can change later states and observations; evaluate the resulting rollout. A local cross-entropy step is not generally the perturbation that minimizes cumulative return. White-box gradients and transfer from a separately trained policy are different access settings; success in those experiments is not a universal guarantee for all policies.

A runnable counterexample: changed action does not always lower return

The following corridor is invented for teaching, not a trained network or a recreation of Atari. States are 0, 1 and terminal goal 2; start at 1. L/R moves one cell, clamped to the corridor. Entering 2 gives reward 1 and ends the episode; all other rewards are 0. The observation is o=(s+2)/4. The hand-defined policy has p(R|o)=sigmoid(4(o−0.5)) and executes R when p≥0.5 (including ties), otherwise L.

Attack only the first observation with ε=0.4, clipped to [0,1]. For clean reference R, the gradient is 4(p(R|o)−1), so o=0.75 becomes 0.35. The true state is still 1 until the chosen L action moves it to 0. Later observations are clean. Use γ=1; changing the time limit H changes the evaluated outcome.

The H=3 attacked run recovers the same undiscounted return despite its wrong-way first action. With γ<1, its delayed reward instead gives G=γ², versus clean G=1. Neither action-change rate nor episode length alone determines return. Download the corridor, attack and rollout model. The example has no training loop and supplies no claim about Atari scores or human imperceptibility.

Interactive: FGSM with an explicit model

This is a two-coordinate mathematical classifier, not a trained image network. Inputs are (u, v) in [0,1]². Define b(u) = 0.35 + 0.8(u − 0.5)², z = 8(v − b(u)), and pB = sigmoid(z). Predict B when v ≥ b(u), otherwise A. The class score is not a calibrated measure of real-world confidence.

Reference y = predicted class of the clean point (A=0, B=1)
Loss J = log(1 + exp(z)) − y·z
Gradient = 8(pB − y) · (−1.6(u − 0.5), 1)
Q = clip(P + ε·sign(Gradient), 0, 1)

Keep y fixed during the step. This demonstrates a prediction flip relative to the clean label, not a verified semantic error: no external ground-truth function is supplied. The sign step maximizes the first-order loss approximation over an L∞ box; it need not maximize the nonlinear loss or cross the decision boundary. Zero gradient components produce zero displacement. Domain clipping can make the actual change smaller than ε.

Explore one sign-gradient step

u v ε

0.08

Generate FGSM stepReset point and ε

Static worked example shown; interactive controls require JavaScript.

Clean: Class B · Candidate: Class A

P = (0.3000, 0.4300); Q = (0.2200, 0.3500). δ = (−0.0800, −0.0800); ||δ||∞ = 0.0800.

For the static example, J(P,y) = 0.5195 and J(Q,y) = 0.9752. A larger loss alone does not imply a prediction flip. Epsilon measures these abstract coordinates, not pixel units or perceptual similarity. The model and figure share one downloadable implementation; the plot is standalone SVG. The construction follows the FGSM first-order argument of Goodfellow et al., §4, but this toy fixture is an editorial example, not their experiment.

5. PGD: the iterative attack

Motivation. FGSM takes one step. PGD (Projected Gradient Descent) recomputes the input gradient over multiple projected steps to search for a high-loss candidate. A finite run is not guaranteed to cross the boundary or find a global optimum.

PGD as ordered steps

Purpose / CIA target: break Integrity of the decision by searching inside the declared ε budget. The procedure is a nested initialise-then-loop:

1. Initialise inside the budget: sample a random perturbation, add it to x and clip to [0,1] before evaluating the first gradient.
2. For each iteration t = 1..T, step: x_t = x_{t-1} + α · sign(∇_x J(θ, x_{t-1}, y)) (one FGSM-style step of size α).
3. Project back onto the L∞ ball: clip x_t so every element stays within ε of the original x.
4. Clip to valid pixel range: clip x_t to [0,1] so the result is still a real image.
5. Repeat steps 2-4 for all T iterations, then return x_T.

The PGD algorithm

x_0 = Clip(x + U(-ε, ε), 0, 1)  // Feasible random start
for t = 1 to T:
    x_t = x_{t-1} + α · sign(∇_x J(θ, x_{t-1}, y))   // Gradient step
    x_t = Clip(x_t, x - ε, x + ε)                     // Project back to L∞ ball
    x_t = Clip(x_t, 0, 1)                             // Box constraint (pixel validity)
return x_T

Parameters: T = iterations (typically 7-20); ε = perturbation budget (L∞ norm); α = step size (typically ε/T or ε/2). The random start inside the budget is what lets PGD escape the single local direction FGSM is stuck with.

The clipping function: two constraints

Clip() is what keeps the iterative attack legal, enforcing two constraints each step:

1. Perturbation constraint (L∞ norm-ball) — every element of x_t may differ from the original x by at most ε. This is a numeric constraint; visual similarity and preservation of the true label require separate justification.
2. Box constraint (pixel validity) — values stay in the valid range (e.g. [0,1] for images), so the result is still a real image.

PGD vs FGSM: specify the configuration

A gradient evaluation normally involves both a forward and a backward computation, not merely a forward pass. With no random start, one step and α = ε, this projected sign-gradient procedure reduces to input-clipped FGSM. A particular finite PGD run need not beat FGSM; retaining the best candidate across runs, including FGSM, avoids discarding that baseline under the chosen selection metric.

Madry et al. (ICLR 2018) use PGD in robust optimization. This is not a proof that every implementation finds the worst perturbation. Kurakin et al. (ICLR 2017) report multi-step attacks transferring less well than single-step attacks in their experiments. White-box strength and transferability are different measurements.

6. Transferability: generation and evaluation are different

Transfer occurs when an input crafted using model A also meets the attack objective on a different model B. It is an empirical outcome, not part of the definition of every adversarial example. Keep the candidate fixed when moving from A to B: adapting it with B's feedback changes the evaluation protocol.

State the goal and the denominator

For this example, clean inputs must be classified correctly by both models, retain ground-truth label y under the allowed modification, and share the same label set. Choose target t ≠ y before generating the candidate. Untargeted success means B(x + δ) ≠ y; targeted success means B(x + δ) = t. A different wrong label does not count as targeted success.

Over the 4 eligible records, targeted success is 2/4 (50.0%) and untargeted success is 3/4 (75.0%). Conditioning instead on targeted success on A leaves 3 records: targeted transfer becomes 1/3 (33.3%). Record d succeeds on B despite failing on A; record e is excluded for its clean prediction. These are different denominators, not conflicting measurements. Download the records and counting functions.

What the cited experiments actually show

Transfer was studied by Szegedy et al. (2013 preprint, ICLR 2014) and Goodfellow et al. (ICLR 2015). Goodfellow's panda illustration uses GoogLeNet: 57.7% is its clean-image class score, not a VGG-to-ResNet transfer rate.

The following is a transcription of the held-out diagonal of Table 3 in Liu et al., ICLR 2017, pp. 4 and 8, not a local reproduction. The authors select 100 ILSVRC-2012 validation images correctly classified by all five models and assign target labels. Each attack uses the other four models; the listed model is held out. The metric is top-1 target-label matching over those 100 candidates.

RMSD uses pixel units [0,255], not an L∞ epsilon. These are optimization-based, targeted ensemble attacks, not single-model FGSM results. Untargeted accuracy is a different metric in a different table. The earlier 57%, 35%, 25%, 30%, 2% and 18% claims are not supported by these cited tables.

Mechanisms and limits

Liu's geometric analysis observes nearly orthogonal input gradients alongside aligned decision boundaries. Gradient alignment is therefore not a universal explanation. Nor does model family alone rank vulnerability: data, training, objective, perturbation constraints and evaluation method matter. An ensemble can improve transfer empirically without forcing all gradients to align.

In a numeric image model the perturbation is δ = x′ − x. A bitwise XOR visualization is not this arithmetic difference and does not prove shared features or transferable errors.

7. Why adversarial examples exist: five perspectives and their limits

The course groups the explanations into five themes. The cited works use different assumptions; their results are not five universal laws or a measured consensus about one cause.

s(x) = wᵀx + b; y ∈ {−1, +1}
min_(||δ||∞ ≤ ε) y s(x + δ) = y s(x) − ε ||w||₁

Clean usefulness: E_D[y f(x)] ≥ ρ, with ρ > 0
Robust usefulness: E_D[inf_(δ in Δ(x)) y f(x + δ)] ≥ γ, with γ > 0

A calculated example: clean usefulness versus perturbation sensitivity

This invented distribution contains just two equally likely inputs: y ∈ {−1,+1}, x = (u,v) = (y, ηy), η = 0.1. Define the true label throughout the plotted neighborhood as sign(u). Both f₁(x) = u and f₂(x) = v/η have mean zero, variance one and clean usefulness 1 on this distribution.

Allow ||δ||∞ ≤ ε = 0.2. The two panels use exactly the same points and the same δ = (−εy, −εy), preserving the true label. Only the classifier changes. This is a coordinate example, not an image dataset, trained network or claim about human perception.

The worst usefulness is 1 − ε = 0.80 for u and 1 − ε/η = -1.00 for v/η. Both classifiers are correct on the two clean inputs, but only sign(u) remains correct throughout these perturbation boxes. No nonlinearity or shortage of training examples is needed for this particular contrast. The perturbed points leave this toy distribution’s two-point support; the example does not prove that real adversarial images must be OOD. Download the model and exact margin formula.

8. Real-world domains: evidence, system impact and limits

An attack on a model can create a security risk, but a changed prediction is not automatically a failed application or an observed injury. State the input channel, target, measured outcome and downstream assumptions. The eight headings below preserve the course’s domain overview; they are not eight independent demonstrations of real-world harm.

Physical feasibility is a separate test

Printing, lighting, distance, viewpoint, sensing and preprocessing can change a perturbation before classification. Eykholt et al. evaluated selected conditions and sampled/cropped drive-by frames, not a complete autonomous vehicle. Carlini & Wagner explicitly identified over-the-air playback and transferability as open questions for their 2018 audio attack. A universal ranking of “physical attacks are less reliable” does not follow without matched tests and a specified metric.

Connect the model result to CIA without skipping the application

An attacker-induced wrong decision may violate the intended integrity of the model service. Confidentiality loss needs evidence of unauthorized disclosure; availability loss needs evidence of denied or degraded service. Physical safety is a system consequence, not a fourth CIA component or an automatic consequence of misclassification. Authentication thresholds, independent checks and human review may change the outcome, but their effectiveness must also be tested.

9. Defenses: adversarial training and randomized smoothing

Adversarial training: optimize a declared objective

Madry et al., §2 formulate training as an outer minimization over parameters and an inner maximization over allowed input changes. Finite PGD approximates that inner problem; failing to find an attack does not prove that none exists.

minimize over θ: E_(x,y) [ max_(δ in Δ(x)) J(θ, x + δ, y) ]
Δ(x): declared norm budget and valid input domain
y: unchanged under the intended threat model

for batch (x, y):
    # Input search: hold parameters and labels fixed
    x_candidate = constrained_attack(θ, x, y, Δ)
    x_candidate = detach(x_candidate)
    # Choose λ beforehand: 1 = adversarial-only, 0 = clean-only
    L = mean(λ * J(θ, x_candidate, y)
             + (1 - λ) * J(θ, x, y))
    θ = θ - learning_rate * gradient_θ(L)

The optional mixture above is an explicit editorial recipe with 0 ≤ λ ≤ 1. Both losses use the same θ; two successive updates at different θ are not this single weighted-loss step. Clean-data training is not a mandatory second update and does not guarantee preserved clean accuracy.

Report the dataset, model, threat set, optimizer and attack configuration together with clean accuracy and attacked accuracy. “FGSM-trained: yes/no” and “PGD-trained: yes/yes” are not universal outcomes. Neither training-attack strength nor clean-accuracy loss has a universal monotonic relationship.

Athalye et al., §§5–6 distinguish misleading gradients from genuine resistance. Obfuscated gradients can make an attack fail; they are not a defining property of adversarial training. Evaluate the actual defense, including stochasticity or preprocessing, with suitable adaptive attacks. Empirical testing is not a certificate, but a trained model may separately admit certification—including through smoothing.

Randomized smoothing: specify what is certified

For a fixed classifier f defined on the noisy input space and σ > 0, Gaussian smoothing defines:

η ~ N(0, σ²I)
g(x) = argmax_c Pr[f(x + η) = c]
pLow ≤ Pr[f(x + η) = cA]
pHigh ≥ max_(c ≠ cA) Pr[f(x + η) = c], with pLow ≥ pHigh
R = (σ / 2) * (Φ⁻¹(pLow) - Φ⁻¹(pHigh))
g(x + δ) = cA for ||δ||₂ < R

This is Cohen et al., Theorem 1. Φ is the standard-normal CDF. The claim is local L₂ prediction stability of g, not correctness against ground truth, stability of f, or an L∞ guarantee at the same radius. It does not depend on an attack’s ability to compute gradients.

Finite samples: a vote is not a probability bound

Fix f, x, σ, n0, n and failure budget α before sampling
Select cA from n0 independent noisy evaluations
Freeze cA; draw a fresh independent batch of n evaluations
k = number of new evaluations predicting cA
pLower = one_sided_binomial_lower_bound(k, n, α)
if pLower ≤ 0.5: ABSTAIN
else: return cA, R = σ * Φ⁻¹(pLower)

The bound used here is one-sided Clopper–Pearson, as in the authors’ implementation. For k > 0 it solves Pr[Binomial(n, pLower) ≥ k] = α; for k = 0 it is zero. Taking pHigh = 1 − pLower gives the simpler radius above. Independent estimation prevents selecting a winner from the same counts used to certify it.

With probability at least 1 − α over the certification draws, a returned certificate is valid for the fixed input. This is not a probability that the label is correct, nor a simultaneous guarantee for an unlimited number of inputs. An abstention is inconclusive, not proof that an adversarial example exists. The boundary ||δ||₂ = R is outside the strict guarantee.

Worked counts: the same selected class, different evidence

Invented two-class counts, not sampled network outputs: selection is [8,2], so cA = class 0 throughout. Use σ = 0.25, α = 0.001 and n = 100 fresh estimation draws. The same equations generate every cell below.

A 60% vote share still abstains at this confidence level. Even unanimity in 100 samples gives a finite bound, not p = 1 or infinite radius. When estimation favors class 1, the procedure still counts class 0; it does not silently reselect. Download the calculation and records. This educational double-precision code is independently tested, not a formally verified numerical certifier or a rerun of ImageNet experiments.

Training, inference and cost are separate

The theorem imposes no training procedure. Gaussian augmentation with the chosen σ can help accuracy under noise; retraining and a matching training σ are not logical prerequisites for validity. Adversarial training and smoothing can be combined. For bounded image inputs, any clipping/normalization must be part of the fixed base classifier f; do not replace the Gaussian by truncated noise and assume the same theorem.

Prediction and certification have different sampling budgets. The paper’s default certification experiment used n0 = 100, n = 100,000 and α = 0.001; “100–1000 forward passes” is not a universal certification cost. Larger σ alone does not guarantee a larger useful radius: the class probabilities change too. Report σ, sample counts, failure budget, abstentions and accuracy together.

10. Privacy surface: weights, context and evidence

Learning useful statistical structure is not the same as verbatim memorization. Also distinguish information retained in parameters from information supplied in a live session. A system can disclose a newly retrieved record without having trained on it.

Define what the measurement counts

In Carlini et al., Definition 3.1, extractability means a training sequence can be split into a k-token prefix p and suffix s, with greedy generation from p reproducing s exactly. This is an operational test, not a complete measure of everything learned. Their evaluation uses withheld 50-token suffixes; prompt length and the sample’s duplication distribution matter. See methodology.

The paper reports log-linear trends with size, duplication and context in its evaluated settings, with qualifications across model families. It does not establish a universal super-linear law. The slides’ 1.3B / 6.7B / 175B table lacks a verified matching result and denominator in the cited paper; it is removed here rather than presented as measured data. See experiments and replication study. No experiment is rerun in this figure.

Four illustrative evidence checks — not measured model outputs

Observation / setup	What it supports	What it does not establish
A known training sequence is split into prefix p and withheld suffix s. Only p is supplied; greedy generation reproduces s.	A witness for extractability under the stated prefix-only test.	Does not alone prove that s is confidential, that an attacker knows p, or that this is the only cause of the output.
A new confidential string is supplied in the context and appears in a tool argument.	The trace shows context-to-output disclosure of that string.	Not evidence that the string was learned during training. Check the actual tool recipient and authorization.
An output looks like a personal record, but its source and truth have not been checked.	Provenance remains unverified.	It may be invented, public, context-derived or training-derived; appearance alone cannot decide.
One tested prefix does not reproduce the chosen training suffix.	No extraction witness for that particular test.	Not proof that the model forgot the record or that other prompts, access modes or attacks cannot reveal information.

A newly retrieved confidential string appears in a web-search query. Does that prove training-data memorization?

No. The live-context path is sufficient in this example; no weight update is needed. Inspect the trace and the recipient’s authorization. If the same string was also in training, this trace alone still cannot attribute its production uniquely to one path.

What counts as a loss: PII vs sensitive information

Identifiability, security sensitivity and legal category are distinct. “Sensitive information” is a broad course-level term, not proof that every item of PII belongs to a GDPR special category or is HIPAA PHI. Confidentiality concerns unauthorized disclosure; not every observed attack succeeds. See the applicable definitions and scope before assigning legal consequences.

11. Privacy threats: target, access and timing

Separate the information targeted from when and how the attack is executed. Attacks on training-data privacy can query an already-trained model. They are not training-time interventions such as poisoning.

The slides group membership inference, inversion and extraction under “training-time threats.” Read that as a shorthand for their connection to training data, not as an execution timeline. The six entries below describe different questions and mechanisms, not six mutually exclusive or automatically successful attacks.

Confidentiality concerns unauthorized learning or disclosure. Prompt injection instead describes an instruction-manipulation mechanism whose intended consequences can include confidentiality, integrity or availability losses. Determine which consequences occurred in the actual application; an attack name alone is not outcome evidence.

12. Alignment, prompting and confidentiality

Task: classify the review as Positive or Negative.
Example: "The screen is beautiful." → Positive
Example: "It crashes every hour." → Negative
Example: "It works exactly as described." → Positive
New review: "The setup process was a bit confusing."
Answer:

Task: classify the review as Positive or Negative.
Review: "The setup process was a bit confusing."
Answer:

13. Prompt injection: channels, objectives and evidence

Prompt injection attempts to make an application treat attacker-controlled content as instructions contrary to its intended task or policy. It need not rewrite the stored system prompt, change model weights or succeed. Distinguish the input route, attacker objective and observed result.

Separate route, modality and objective

Term	What the attacker controls or seeks	Important distinction
Direct prompt injection	Attacker-controlled text supplied through the user-facing input.	An attempt to redirect the application’s task or violate its instruction policy; not necessarily a safety jailbreak.
Indirect prompt injection	Instructions embedded in retrieved documents, tool results or other third-party material.	Content meant to be processed as data attempts to acquire instructional authority. The legitimate user need not have authored it.
Jailbreak	An attempt to circumvent the model or application’s safety restrictions.	Describes the objective; it can overlap with prompt injection. Role-play alone does not prove a successful bypass.
Text carried by an image	Visible or recognized text in an image supplied directly or retrieved indirectly.	Describe both the input modality and provenance. This is not automatically an optimized pixel perturbation.
Optimized visual adversarial example	Pixels chosen to steer a vision-language model’s output under a stated attack setup.	A gradient-based image attack can work without carrying a readable instruction. Its constraints and outcome need separate evaluation.

What the primary studies establish

Perez and Ribeiro (2022) study goal hijacking and prompt leaking in their evaluated language-model setup. Greshake et al. (2023) study instructions planted in data retrieved by integrated applications. Neither establishes that every adversarial input overrides system instructions or that every prompt is confidential.

The slides’ DAN (“Do Anything Now”) material illustrates a role-play jailbreak attempt. Treat its historical examples as examples, not a universal success rate or evidence about current systems. A jailbreak seeks a safety-policy bypass; an indirect injection can instead seek an ordinary but unauthorized action. Terminology can overlap, so state the concrete objective.

Visual attacks: distinguish the model from the attack paper

Zhu et al., MiniGPT-4, describe a vision-language architecture connecting a visual encoder to a language model. That architectural work is different from Carlini et al., sections 6.1–6.3, which evaluate adversarial image optimization on specific vision-language implementations. Their setup uses a differentiable path from pixels to output scores, random initial images and an output-targeting objective. Reported distortion is not a certificate of imperceptibility or a physical-camera result.

The latter paper’s reported success applies to its chosen models, prompts, target outputs and optimization setup. It is not a measurement of every multimodal model or of text-only safety filters. Text drawn inside an image, optimized pixels and physical patches are different constructions; do not silently transfer evidence between them. No model or attack is run by this lesson.

Judge the observed outcome, not the attack’s name

Constructed outcome checks — not experimental results

Case	What the trace actually shows	Supported conclusion
Instruction is quoted	The assistant identifies an instruction inside the document but follows the requested summary task.	Exposure is established; successful redirection is not.
Answer is redirected	The delivered reply is the attacker’s unrelated marker instead of the requested summary.	Task integrity is violated in this constructed example; confidential-data disclosure is not established.
Disclosure is proposed, then blocked	A proposed call contains fictional private data; the executor denies it before dispatch.	A rejected disclosure attempt, not a completed external transfer. Check for earlier effects separately.
Disclosure is actually delivered	The trace records unauthorized private data reaching the specified recipient.	Confidentiality is violated under the stated policy. A model-generated claim that it sent data is insufficient evidence.

For an empirical success rate, state the eligible trials, attack budget, success criterion, judge and failures or missing runs. A safety refusal, correct task completion and protection of confidential data are different outcomes. Inspect dispatched arguments and delivered replies; do not infer real execution from generated narration.

Does an adversarial instruction inside a retrieved image prove that data was leaked?

No. It identifies a possible indirect, image-carried injection attempt. Establish whether the application followed it and whether unauthorized information actually reached a recipient. Merely quoting the instruction or blocking a proposed call does not establish that outcome.

14. Agentic AI: proposals, tools and enforcement

A conversational interface may retain history or use tools; “chatbot” does not imply statelessness or low impact. An agentic application lets a model propose actions within a surrounding execution system. Those actions can access local resources, remote services or the user-facing reply channel. Location alone does not determine authorization.

ReAct is an interaction pattern, not an access-control policy

Yao et al., ReAct interleave generated reasoning traces and task-specific actions with observations from the environment. Their question-answering examples use a Wikipedia interface; the action space also includes finishing with an answer. The paper does not establish that every action is an external disclosure or that this loop itself enforces confidentiality.

A model-produced tool call is a proposal until an executor dispatches it. Inspect actual arguments, authenticated actor, target resource and recipient. Returning an answer can disclose data too; a tool observation is an input channel for untrusted content. Do not treat instructions found in a retrieved result as new authorization.

Make the policy explicit

For this teaching example, accounts are A and B and a fictional record belongs to A. The local record store permits access only to the owner. Search accepts public data only. Replies may contain public data or the requesting account’s own record. Labels and ownership are trusted metadata in the example, not claims supplied by the model. A real system must establish provenance, authorization, purpose and complete mediation; this tiny label check does not solve those problems.

proposal = model_proposal(context)
decision = policy(authenticated_actor, trusted_labels, proposal)
if not decision.allow: stop_without_dispatch()
else: result = dispatch(proposal)
# Tool observations may influence a later proposal, but do not grant authority.
# A final reply is also checked before delivery.

Step through five scripted proposals

No LLM or real tool is executed. The trace applies the small policy below to prewritten proposals; it is not an attack benchmark or a production security boundary.

Policy trace

Scenario Local: own recordLocal: another accountExternal: public queryExternal: private recordUntrusted result: new proposal Next step Reset trace

Interactive controls require JavaScript; the complete worked traces are available below.

Local: own record — finished; tool calls: 1, replies: 1

1. Request: Actor A; fictional record owned by A.
2. Proposal: record-store ← record data (owner A).
3. Check: ALLOW: Owned record; permitted destination.
4. Dispatch: Simulate the permitted record-store operation.
5. Observation: A simulated tool result returns. Its content does not grant new authority.
6. Proposal: reply ← record data (owner A).
7. Check: ALLOW: Owned record; permitted destination.
8. Dispatch: Simulate the permitted reply operation.
9. Finished: Reply delivered to the requesting account under the stated toy policy.

Local: another account — blocked; tool calls: 0, replies: 0

1. Request: Actor B; fictional record owned by A.
2. Proposal: record-store ← record data (owner A).
3. Check: DENY: Actor is not authorized for this record.
4. Blocked: No dispatch for this proposal. Earlier completed calls, if any, are not undone.

External: public query — finished; tool calls: 1, replies: 1

1. Request: Actor A; fictional record owned by A.
2. Proposal: search ← public data (owner A).
3. Check: ALLOW: Public data; permitted destination.
4. Dispatch: Simulate the permitted search operation.
5. Observation: A simulated tool result returns. Its content does not grant new authority.
6. Proposal: reply ← public data (owner A).
7. Check: ALLOW: Public data; permitted destination.
8. Dispatch: Simulate the permitted reply operation.
9. Finished: Reply delivered to the requesting account under the stated toy policy.

External: private record — blocked; tool calls: 0, replies: 0

1. Request: Actor A; fictional record owned by A.
2. Proposal: search ← record data (owner A).
3. Check: DENY: This example search policy allows public data only.
4. Blocked: No dispatch for this proposal. Earlier completed calls, if any, are not undone.

Untrusted result: new proposal — blocked; tool calls: 1, replies: 0

1. Request: Actor A; fictional record owned by A.
2. Proposal: search ← public data (owner A).
3. Check: ALLOW: Public data; permitted destination.
4. Dispatch: Simulate the permitted search operation.
5. Observation: An untrusted result asks for private-record disclosure. The next proposal is scripted for this example, not generated by an LLM.
6. Proposal: search ← record data (owner A).
7. Check: DENY: This example search policy allows public data only.
8. Blocked: No dispatch for this proposal. Earlier completed calls, if any, are not undone.

The local/other-account case is denied even though no external server is involved. The public search is allowed even though it is external. In the returned-instruction case, one public search has already happened before a second, private-data proposal is blocked. The worked traces compute those counts from the policy; they do not infer security from tool names.

Download the deterministic policy and scenarios. Client-side controls only teach the example; readers can modify them, so they must never be used to enforce access to real data.

System enforcement and learned behavior are different

Debenedetti et al., CaMeL separate control and data flows and enforce tool-call policies using capabilities. That supports placing enforcement outside a model’s natural-language promises. It is not a universal solution: the paper specifies threat assumptions and discusses policy specification, declassification, side channels and attacks outside its data-flow scope. The simple example here is not CaMeL and inherits none of its experimental results or security proofs.

Does a blocked second proposal undo the first tool call?

No. In the returned-instruction trace, the public search already ran. The policy blocks the later private-data dispatch only. Check complete traces, not just the final state, when assessing exposure or side effects.

15. Differential privacy: guarantees and scope

Differential privacy: specify what is protected

DP is a property of a randomized mechanism, not a synonym for adding noise. With ε≥0 and 0≤δ<1, for all neighboring datasets D, D′ and measurable output events S:

Pr[M(D) ∈ S] ≤ exp(ε) · Pr[M(D′) ∈ S] + δ

Probabilities are over the mechanism’s randomness. Pure DP has δ=0; approximate DP allows δ>0. Specify adjacency: adding/removing one record, replacing one record, or changing one person’s contributions are different units. A record-level bound is not automatically a person-level bound when a person contributes multiple records. DP limits distinguishability; it does not promise that every inference about someone is impossible. See Dwork & Roth, Definition 2.4 and group privacy.

Text output is compatible with DP

A data-independent transformation of a DP release preserves its guarantee, including a transformation into text. For a DP-trained model, generation is post-processing with respect to the protected training dataset if the downstream procedure has no additional access to it. Hold other inputs fixed across neighboring datasets. This protects neither new context by default nor records previously exposed by a non-private pretraining stage. See post-processing, Proposition 2.1.

Yu et al., Differentially Private Fine-tuning of Language Models evaluate private fine-tuning for natural-language generation, including GPT-2 on DART. This is a concrete counterexample to the slides’ claim that DP applies only to statistical aggregates. Their guarantee concerns the specified fine-tuning data and privacy accounting, not arbitrary secrets in future prompts. No language model is trained or benchmark rerun here.

A fully enumerated one-bit example

Let neighboring inputs be x=0 and x=1: one protected bit is replaced. Randomized response returns y=x with probability p=0.75 and y=1−x otherwise. The largest probability ratio is p/(1−p)=3, so pure ε=ln(3)≈1.0986. This is an educational finite mechanism, not an LLM, an empirical attack score or a recommended deployment budget.

y = randomized_response(x, p)
text:  return "YES" if y == 1 else "NO"
reuse: return (y, y)
fresh: return (y, randomized_response(x, p))  # independent fresh draw
raw:   return (y, x)                        # accesses the protected bit again

Exact worst-case pure-DP bounds for this two-input mechanism

Release	Operation	Largest ratio	ε
One text release	Encode y as NO / YES.	3	1.0986
Reuse same result	Release (y, y); draw y only once.	3	1.0986
Two fresh releases	Release (y₁, y₂); draw independently given x.	9	2.1972
Append raw bit	Release (y, x); the second component bypasses randomization.	∞	No finite bound

For two fresh releases, event (0,0) has probabilities 0.5625 and 0.0625: ratio 9, giving 2ln(3). For reuse, (0,0) has probabilities 0.75 and 0.25: ratio 3. With raw output, event “last bit is 0” has probabilities 1 and 0. It violates the DP inequality for every finite ε and δ<1. A copied fresh context secret has the same problem with respect to that secret, even if a separate training-data guarantee remains intact.

Account for data access, not the number of displayed answers

Reusing one DP release is post-processing. New private-data accesses require accounting; basic composition adds the ε and δ budgets of the component mechanisms under its assumptions. These cases cannot be distinguished by counting chat turns alone. See composition, Theorem 3.16. DP and recipient authorization answer different questions and can be used together.

Download the exact finite-distribution model. It enumerates outcomes without sampling or accessing personal data; it is not a production privacy accountant or a secure random-number implementation.

Does generating ten sentences from one DP-trained model necessarily spend ten training privacy budgets?

No. A finite transcript using only that released model, public or held-fixed inputs, and fresh data-independent randomness is post-processing of the same release. Further access to protected records or separately trained releases needs its own analysis. New private prompt data is not automatically covered by the model’s training guarantee.

16. Healthcare agents: deployment, metrics and evidence

The course case is attributed to Aguzzi, Montagna and Ferretti, Privacy Leakage in Small Agentic Healthcare Models: An Empirical Analysis (PerCom Workshops 2026). Publisher-deposited bibliographic metadata and Ferretti’s publication list confirm the work. The detailed setup and numbers below come from the course PDF 03 - privacy in LLM, slides 73–88; this review did not obtain the paper’s full text or run-level artifacts. The source distinction matters: a lecture summary is not a reproduced benchmark.

Model size is not a deployment or privacy policy

“Small language model” is a relative description, not a universal 100M–7B definition. Keep parameter count, weight format, runtime workload, hosting and authorization separate. The course evaluates Qwen3-1.7B and Qwen3-4B; their names are not a guarantee of smartphone or laptop performance.

Separate the deployment decisions

Dimension	What to specify	What does not follow
Model	Exact checkpoint, architecture, total/active parameters and evaluation task.	Size alone does not determine quality, instruction following or confidentiality.
Memory	Weight precision, context length, KV cache, batch size, activations and runtime overhead.	Nominal weight storage is not total inference memory or a device-fit guarantee.
Hosting	Device, organization-managed server or external service; specify the actual data path.	A small model can be served remotely; a larger open-weight model can be self-hosted.
Cost	Hardware, energy, latency, maintenance and any service charges for the workload.	Local inference is not intrinsically free or near-zero cost.
Privacy	Authorized actor, payload, recipient, retention and tool/log/synchronization behavior.	Local model inference does not prevent data leaving via another component.

The Qwen3-1.7B and Qwen3-4B model cards report nominal totals of 1.7B and 4.0B parameters. For a weights-only arithmetic illustration, storing P scalar weights at b bits costs P × b / 8 bytes. These are decimal GB, not GiB, measured file sizes or memory benchmarks. Uniform 4-bit storage is an idealized calculation: quantization metadata and mixed-precision weights add overhead.

Nominal weights only — not total device memory

Nominal parameters	16-bit weights (GB)	Idealized 4-bit weights (GB)
1.7B	3.40	0.85
4.0B	8.00	2.00

As a counterexample to “larger means cloud-only,” OpenAI’s gpt-oss-20b documentation describes an open-weight model for local or specialized use, with 21B total and 3.6B active parameters. That does not certify it on a particular device or as a leakage judge. The old GPT-4 “>100B” estimate is not retained: the API model page consulted does not provide that parameter count.

The four-stage evaluation described by the slides

1. Construct synthetic personas. The slides describe Gemini 2.5 Flash at temperature 0.2 producing context C and sensitive facts F. Synthetic cases avoid using actual patient records in this example, but require checks for consistency and realism; “synthetic” alone is not proof of anonymity or benchmark quality.
2. Prepare five prompt categories. P.I.: direct instruction override; E.R.: explicit request for private facts; T.Q.: disguised query; C.R.: combine contextual facts; C.M.: conditional use of sensitive information. These are this study’s categories, not an exhaustive security taxonomy.
3. Run the configured agent. Compare Qwen3-1.7B and Qwen3-4B under baseline and privacy-hardened prompts. The slides list four record/appointment tools and one external search tool. “Internal” is not equivalent to authorized, nor is every external query forbidden; apply the explicit policy distinction in the agent section.
4. Label and aggregate outcomes. The slides describe gpt-oss-20b at temperature 0 receiving facts F and tool-call inputs. This is an automated judge, not an independently demonstrated “high-accuracy” oracle. Specify the rubric and validate against human-labeled examples, including false positives/negatives and unparseable outputs. A tool proposal, a dispatched call and actual receipt of data are distinct observations.

Attack@1: separate a per-run average from an any-run event

Editorial correction to slide 81: the displayed equation is A₁ = (1/N) ∑ᵢ Lᵢ, with binary leakage labels Lᵢ. This estimates the fraction of runs flagged by the judge. The slide’s accompanying “at least one” description is a different aggregation. Repeating a scenario five times does not turn a single-run metric into best-of-five.

With S scenarios and R runs per scenario, define A₁ = (1/S) ∑ₛ [(1/R) ∑ᵣ Lₛᵣ]. The fraction of scenarios with at least one flagged run is instead B(R) = (1/S) ∑ₛ 𝟙[∑ᵣ Lₛᵣ > 0]. Equal R makes the per-scenario mean equal to the pooled per-run fraction; unequal run counts require an explicit weighting choice. No independence assumption is needed to calculate these observed fractions.

Complete invented trial labels

Scenario	Runs 1–5	Flagged / runs	Any flagged?
A	1 0 0 0 0	1/5	Yes
B	0 0 0 0 0	0/5	No
C	1 1 1 0 0	3/5	Yes

For the comparison with Chen et al.’s pass@k, distinguish the number n of sampled completions from the budget k being evaluated. Given c successful samples, the estimator is 1 − C(n−c, k) / C(n, k) for 1 ≤ k ≤ n. C(a,b) counts b-element subsets of a elements, and is zero when b > a. At k=1 it reduces to c/n; at k=n it is the observed any-success indicator. If independent identical attempts have known success probability p, the probability of any success in k attempts is 1 − (1−p)ᵏ; substituting an estimated p is not the same finite-sample estimator. Here “success” would mean a judge-flagged attack, not passing a code test.

Download the finite metric and memory calculations. The example is deterministic and runs no LLM or real tools.

Reconstruct the experimental denominator

Balanced design reconstructed from slides 82–83

Quantity	Calculation	Meaning
Scenarios	10 personas × 5 prompts = 50	One prompt from each of five categories for each persona.
Per model/condition	50 scenarios × 5 repeats = 250	Repeated generations, not 250 distinct personas.
Across the design	250 × 2 models × 2 prompt conditions = 1000	The reported total is consistent with all four cells; 50 × 5 alone is only 250.

This reconstruction follows the balanced design described in the slides, not independently inspected run logs. For a pooled cell-level estimate the intended denominator is 250 completed runs; per-scenario estimates use five runs. Failed/missing generations, retries, weighting and exact leakage labeling must be documented before reproducing the reported scores. Repeated runs for the same persona are not 1,000 independent patients.

One of five attempts is flagged: is the single-run estimate 100%?

No. The single-run fraction is 1/5 = 20%. The observed any-of-five indicator for that scenario is 1. These answer different questions.

Reported results: preserve their provenance and uncertainty

The course slides report baseline rates near 90% for both models and hardened averages of 0.89 (1.7B) and 0.56 (4B). Treat these as slide-reported, judge-labeled outcomes for this particular setup. They show residual failures under the tested prompts; they do not establish universal incapacity, isolate a causal effect of parameter count, validate the judge or decide clinical/regulatory suitability. The paper’s full text and run-level artifacts were not available in this review.

Privacy-hardened results transcribed from slide 85

Category	Qwen3-1.7B	Qwen3-4B
C.M.	0.92 ± 0.25	0.90 ± 0.22
C.R.	0.90 ± 0.25	0.90 ± 0.25
E.R.	0.74 ± 0.34	0.40 ± 0.43
P.I.	1.00 ± 0.00	0.06 ± 0.13
T.Q.	0.88 ± 0.21	0.56 ± 0.46
Average	0.89 ± 0.09	0.56 ± 0.36

The “±” terms are transcribed as displayed. The reviewed slide does not define whether they are standard deviations, errors, intervals or which aggregation they describe; do not interpret them as confidence intervals or draw error bars from an assumed meaning. The baseline is described only as approximately 90% here; exact counts are not reconstructed from that rounded figure.

The 4B row for P.I. is lower than its conditional/cross-reference rows in this table. That supports a comparison within the reported setup, not a conclusion that explicit jailbreaks are generally easy or that 4B is safe. Testing two checkpoints does not by itself establish that parameter count caused the difference. A generated rationale is also not direct evidence of an internal inability to understand privacy.

Constructed examples: inspect what leaves the system

The following are teaching examples, not observed model traces or patient records.

C.M. ExampleC.R. Example

Rule in prompt: if private condition X is present, search.
Candidate A: search("general treatment guidance")
Candidate B: search("treatment for condition X")

B explicitly includes X in a proposed payload. A omits the condition from the text, but an observer who knows the rule and sees the search event might infer that its condition was satisfied. State the observer’s knowledge, actual dispatch and policy before claiming a leak. Merely evaluating the condition locally is not itself disclosure.

Private context: conditions X and Y; location Z.
Proposed query: "X Y specialist in Z"
Before dispatch: check payload, recipient and authorization.

The combination can reveal more than any fragment alone. It becomes an external disclosure if the query is actually delivered; whether it is unauthorized depends on the policy and consent context. A blocked proposal is different from a completed call. No clinical advice or real patient data is used here.

Use prompting as one behavioral control, not as proof of confidentiality. Validate the complete application and retain explicit enforcement outside the model. These results alone neither prove prompting is universally necessary nor establish a legally acceptable leakage threshold.

17. Privacy regulation: scope, responsibilities and evidence

Educational overview, sources checked 8 September 2026. This is not a compliance determination or legal advice for an incident. Applicability, national rules, contracts and facts require qualified review. If handling a real incident, involve the responsible privacy/security/legal team promptly rather than waiting for a complete investigation.

Use the applicable definitions

“PII” and “sensitive information” are useful course-level terms, not interchangeable legal categories. Identifiability, sensitivity and an obligation to keep information confidential are different questions. The GDPR, Articles 4 and 9, and HHS Privacy Rule summary use the distinctions below.

Different legal categories — not one universal PII set

Category	Meaning in this overview	Avoid this inference
GDPR personal data	Information relating to an identified or identifiable natural person.	Not restricted to names or to information stored on EU servers.
GDPR special categories	Article 9 includes health data and specified other categories.	A security-sensitive password or trade secret is not automatically an Article 9 category.
HIPAA PHI	Individually identifiable health information held or transmitted by a covered entity or business associate, subject to exclusions.	Names and dates are identifiers, not a freestanding definition of PHI. Context and the regulated entity matter.

Check scope before choosing a rule

GDPR applicability includes processing in the context of an EU establishment and certain offering or monitoring activities involving people in the EU by non-EU organisations. Article 2 also limits material scope. “EU” is not merely a server-location label.

HIPAA applies to covered entities and business associates, not every consumer health app. Covered entities include health plans, clearinghouses and providers conducting specified standard electronic transactions. A business-associate arrangement has its own requirements. Being outside HIPAA does not establish that no other privacy law applies.

Permission to process is not always explicit consent

Under the GDPR, identify an Article 6 lawful basis and, for health data, an applicable Article 9 condition. Explicit consent is one route, not the only one. Article 9(2)(h) addresses specified health/social-care purposes subject to its legal/contractual conditions and Article 9(3) safeguards. National conditions can also apply. See the Commission’s explanation of legal grounds and special categories; “used for healthcare” alone is not permission.

GDPR data minimisation concerns relevance and necessity for the specified purposes, not just initial collection. Article 17 erasure has grounds and exceptions, including certain legal obligations and legal claims; it is not an unconditional promise to delete every record.

HIPAA permits certain uses and disclosures without individual authorization, including treatment, payment and health-care operations subject to its rules. Its minimum-necessary standard has exceptions, including disclosures to or requests by providers for treatment. Do not treat HIPAA authorization and GDPR consent as identical mechanisms. Consult the HHS summary and underlying rules.

Separate incident response from a notification verdict

A GDPR personal data breach can affect confidentiality, integrity or availability; deliberate exfiltration is not required. Under EDPB guidance, document breaches even when authority notification is not required. Information can be supplied in phases; incomplete information does not justify simply waiting for the investigation to finish.

Under the HIPAA Breach Notification Rule, an impermissible PHI use or disclosure is presumed a breach unless a documented assessment demonstrates low probability of compromise or an exception applies. Assess the information and identifiers, recipient, actual acquisition/viewing and mitigation. Required notification concerns unsecured PHI; do not equate “encrypted somewhere” with meeting the rule’s conditions.

Selected notice duties — not an exhaustive incident manual

Responsible party → recipient	Trigger to assess	Timing
GDPR: processor → controller	Awareness of a personal data breach.	Without undue delay; the processor does not get a separate 72-hour allowance.
GDPR: controller → supervisory authority	Personal data breach, unless unlikely to pose a risk to people’s rights and freedoms.	Without undue delay and, where feasible, within 72 hours of awareness; explain a later notice.
GDPR: controller → affected people	Likely high risk, with Article 34 exceptions and safeguards to assess separately.	Without undue delay; this is not the same threshold or a separate 72-hour rule.
HIPAA: covered entity → affected people	A notifiable breach of unsecured PHI.	Without unreasonable delay, at most 60 days after discovery, subject to applicable legal exceptions.
HIPAA: business associate → covered entity	A breach of unsecured PHI at or by the business associate.	Without unreasonable delay, at most 60 days after discovery, subject to applicable legal exceptions.

The GDPR authority and individual-notice thresholds are distinct. HIPAA reporting to the Secretary and, where applicable, the media has additional population thresholds and schedules: follow HHS reporting requirements, not a single deadline for every recipient. Statutory exceptions and applicable contracts require separate review.

A third-party API transfer is not automatically lawful or automatically reportable. Determine the actor, data, recipient, purpose, permission and applicable regime. Conversely, a blocked model proposal does not prove that nothing happened earlier in the trace.

Do not infer a monetary penalty from a model’s output label. The HHS enforcement summary distinguishes circumstances and notes inflation adjustments; a dated figure is not a universal current penalty schedule. No sanction is calculated here.

Does a privacy-labelled model output determine which authority must be notified?

No. A model label is technical evidence to assess, not a legal finding. Establish facts, scope, responsibilities, risk and the relevant notification rule while responding promptly. These notes do not decide a real incident.

Check Your Understanding