How model output becomes XSS, commands, or data exfiltration
Identify the consumer before validating output. HTML, URLs, and operation arguments require different controls; valid JSON, escaping, and authorization are separate properties.
All posts
Page 6 of 128
Learning AI Security: Practical Paths Through Attacks, Defense, and Investigation
Follow one support assistant through AI-assisted vulnerability discovery, protection of AI data and tools, and AI-assisted investigation and remediation. Each path produces reproducible evidence.
What Must You Verify After an AI Patch?
A patch must block unauthorized access, preserve legitimate behavior, and pass against the actual target. A cleared alert and a passing local copy each establish only part of that result.
How AI attacks another AI: indirect injection and tool misuse
Trace hostile attachments into tool requests and separate model susceptibility from backend data exposure using a live-LLM training lab and HTTP replay.
A correct RAG answer can still disclose someone else’s data
Authorize every retrieval before data reaches the model. Cache hits and post-revocation responses must also respect current access to their source data.
Security Assistant Tools: From Read-Only Investigation to Response
Investigation access does not imply response authority. A trusted executor must verify role, incident, exact proposal, and current state before applying a change and recording its result.
Agent sandboxes, credentials, and network boundaries
Probe OS-enforced file and network denials separately from scoped, expiring authorization. Application allowlists are not OS sandboxes, and transport choices do not restrict process capabilities.
How to verify defenses: red teaming, regression, and monitoring
Replay an independent authorization matrix against real HTTP responses before and after a fix, correlate audit events, and report model behavior, enforcement, and production monitoring separately.
An available tool does not authorize this invocation
Bind a ticket export to its caller, resource, action, destination, and approval. Test substituted arguments, revocation, expiry, and replay.
AI Says It Found a Vulnerability. What Evidence Do You Need?
Treat AI suggestions as hypotheses. Preserve actual inputs and outputs, check the target data against the authorization policy, and use normal and patched controls to rule out false positives.
Let AI look for vulnerabilities: from clues to testable hypotheses
Turn AI suspicions into falsifiable authorization hypotheses with same-tenant and cross-tenant controls.
How Can You Demonstrate That AI Helps Security Work?
Freeze independent labels before measuring false positives, false negatives, abstention, human effort, and unsafe actions. Faster closure does not establish correct closure, and rule replay is not model evidence.
Chunking and Indexing: How Big to Cut, What to Search With, and How to Know You Chose Right
Building an index means three decisions: the chunking unit, whether to search with keywords, vectors or both, and which embedding model to use. All three are semi-permanent, since changing any of them means rebuilding. The research is cautious: semantic chunking isn't consistently better, splitting by sentences or by document structure is usually enough, and overlap has small, setup-dependent effects. In one Traditional Chinese study, the correct passage ranked first 57% of the time with vector search alone, keyword search 76%, and the two combined 81%. Don't choose by feel: compare two setups on order 11's test questions, changing one thing at a time.
Evaluation and Launch: How to Know Your AI Knowledge Base Works, and Keep It Improving
Evaluate in two layers: did retrieval find the source passage, and is the answer grounded in it and correct. Looking only at answers can fool you: one study found that when the context wasn't enough to answer, models still got 35% to 62% right using their own knowledge, which breaks on questions only your organization can answer. Use order 11's test questions for both layers, and evaluate permission-crossing and unanswerable questions too; calibrate LLM judges against human samples. After launch, turn escalated and complained-about conversations into failure cases, hand them back to document owners, and add them to the test set.
Structure and Metadata: Which Fields to Record Besides the Text
Metadata serves three purposes: returned for citation, used in search, and used to filter before retrieval; exact conditions like model, year or region belong in fields, and one study that wrote the company and year in front of every chunk raised top-five hits from 33% to 55%. Take filter values from source systems as fixed options, store dates as numbers, and don't let an LLM generate them freely.
Parsing and Cleaning: What Quietly Breaks When PDFs, Tables and Scans Become Text
When documents become text, what breaks usually doesn't throw an error: multi-column paragraphs joined in the wrong order, merged table cells flattened, headers and footers mixed into the body, OCR typos in scans, images and strikethroughs gone. High OCR accuracy doesn't mean good RAG either: in one study, documents with strikethroughs and color marks scored 82.9% on OCR but only 52.8% on question answering. Chunking matters more than switching tools: from the same conversion output, chunking by heading hierarchy with the heading path attached raised table questions from 66.6% to 99.8%. So always spot-check parsed output against the originals for these failure types.
Permissions: How to Make Sure the AI Doesn't Hand Department A's Documents to Department B
Filter permissions at retrieval time: each chunk carries the groups allowed to see it, each query carries the user's groups, and anything that doesn't match never becomes a candidate, rather than letting the model read it and then telling it not to say it. AI doesn't create new permissions, but it makes documents that were shared too widely findable with a single question, so clean up oversharing before launch. When permissions change, the index doesn't necessarily follow: some connectors update permissions only on full crawls, up to a day late. Filtering has a cost too: with strict conditions, some vector databases return too few results. Before launch, add a permission-crossing question to your test set.
Security: Can Documents in Your Knowledge Base Leak Personal Data or Give Orders to the AI?
Documents in a knowledge base may carry personal data or hidden instructions for the AI, and the latter has been demonstrated on Slack AI and Microsoft 365 Copilot. Telling the model to ignore them in the system prompt does little; it's safer to add personal-data and instruction gates at ingestion and limit what the model can do. No detection tool catches everything, so every layer only reduces the risk.
Inventorying Sources: Which Documents Belong in an AI Knowledge Base, and Which Version to Trust
Importing the whole shared drive is the most common way to start an AI knowledge base, and the most common mistake. When old and current versions are retrieved together, the model blends them: in one study, even with the current information retrieved, adding a single outdated passage cut the overall score by more than 24%. First build a source inventory recording each source's owner, update frequency, access restrictions and handling decision; collect only one workflow's well-governed sources in the first batch; and keep only the current version of each document, labeling any old version you must keep with its version and expiry date. After importing, check the file counts, because some connectors import nothing and report no error when they lack permissions.
Updates and Deletion: How an AI Knowledge Base Keeps Up When Documents Change or Disappear
When documents change or are deleted, the index doesn't keep up on its own. A sturdier incremental sync uses a content hash to find which documents changed and redoes only those; deletion has to reach every place a document is stored, including the full-text index and the vector store. Many tools don't handle deletion by default, and deletes and updates can take a while to take effect. Why clean up thoroughly: one preprint put a single outdated document into the context, and two small models flipped three to four in ten medical questions they had answered correctly; when told to answer from the documents, that rose to between two-thirds and three-quarters.