Browse the book & resources

Chapter 13 / 20

13. Security & Compliance: What Is the Cost of a Mistake?

Provisional draft. These short chapters are not approved for publication. Original practitioner interviews, deeper reporting and author review remain outstanding.

An assistant that suggests a test is not the same risk as an agent that can read customer records and modify production. Both may be called "AI adoption" on a dashboard. The permissions, review, and evidence required should be different.

Start with the action and its worst plausible failure. What data can the system see? Can untrusted content redirect its instructions? Can it create a PR, merge it, deploy it, or send data out? Who can stop or reverse the action, and what would a rollback leave untouched? OWASP's LLM application risks include prompt injection, sensitive-information disclosure, and excessive agency; this is a threat taxonomy, not a measured incident rate at your company. NIST's generative AI profile offers a risk-management lens, not a universal ban.

A good boundary makes low-risk work easy and high-consequence work inspectable. Let a drafting assistant work with approved context; sandbox an agent's tools; require evaluation on representative adversarial cases and an authorized human before consequential writes. Keep action logs that let responders reconstruct what happened without treating every employee's prompts as a surveillance feed. Review the full path, including dependencies and the chance that an apparently harmless summary becomes a decision input.

Security also has an opportunity cost. A late, opaque approval gate can strand safe changes; an unrestricted fast lane can transfer risk to customers. Pair security findings and exposure with approval delay, rework, and customer trust. Automating vulnerability triage may let a security engineer focus on severe issues, but only if false positives, false negatives, and who verifies patches are counted.

At the junction: For one proposed agent, list every read and write permission, the source of untrusted input, the irreversible action, and the human who owns the result. Remove unnecessary authority before asking reviewers to accept more generated work.

Sources cited in this chapter

  1. https://genai.owasp.org/llm-top-10/
  2. https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence

Browse all references →