Password Password

Security, explained for the people who have to fix it.

LLMs and Security

LLMs in the SOC: what the papers measured for alert triage

Ten 2026 papers on language models in security operations, read for their numbers, from triage misses to log summaries an attacker can steer.

Put a language model on a security team's alert queue and the number you get back depends less on the model than on how much of the investigation it has to do by itself. Hand it one alert and the packet capture behind it, and the best of four models in one 2026 benchmark labels 98 of 100 alerts correctly. Make it fetch its own evidence from a SIEM, and in another 2026 benchmark the best of five triage designs still closed 40.4% of the attack alerts as benign. Ask it to hunt through raw Windows event logs with no hints, and the best model in a third benchmark flagged 3.8% of the malicious events. This article lines up what ten recent papers measured, rebuilds one benchmark's confusion matrix so a 40% miss rate turns into a count of alerts, and ships a script that scores your own pilot the same way.

How we read the record

We searched the OpenAlex index for arXiv papers published since June 2024 on language models in alert triage, log analysis and security operations, then read each candidate's abstract page and, where arXiv offered an HTML version, the full text, on 14 October 2026. Ten papers made the cut: the ones that report a measured number on a security operations task, or measure the analysts who do the work. All ten are preprints, and arXiv does not referee what it posts. In several, the authors evaluate a system they built themselves; the dataset marks those rows. Each figure below is quoted from its paper, and the quote ships in the CSV beside it.

Three settings, very different numbers

What the model had to do Paper Best reported result
Label one alert, with its packet capture supplied SIABench (arXiv:2603.06422) GPT-5: 98% accuracy on 100 alerts; GPT-4o-mini: 75%
Answer forensic questions over pre-filtered, retrieved logs Retrieval-augmented analysis (arXiv:2603.18196) Claude Sonnet 4: 94% average recall over 17 malware-traffic scenarios
Investigate replayed cloud incidents SIR-Bench (arXiv:2604.12040) the authors' agent: 97.1% true positives detected, 73.4% false positives rejected
Decide what to search in a live SIEM, and when to stop ALERT-BENCH (arXiv:2610.10608) best of five studied designs, GPT-4.1: 40.4% of attack alerts dismissed
Find malicious events in 75,000 to 135,000 raw records with SQL Cyber Defense Benchmark (arXiv:2604.19533) Claude Opus 4.6: 3.8% of malicious events flagged; no model passed every tactic

The gradient runs from curated to open. The curated sets are small and balanced in ways a real queue is not: SIABench's first set is 50 true and 50 false positives, and its second has 5 true positives among 35 alerts, so a single miss moves recall by 20 points. ALERT-BENCH replays recorded enterprise telemetry through a Wazuh SIEM, and its 1,247-alert sample is 225 attack-related and 1,022 benign, which is closer to the shape of a working queue. Of those 225, the paper says 126 are directory enumeration and 94 are WordPress vulnerability scanning: noisy reconnaissance, not a quiet intrusion. The threat-hunting benchmark goes further and gives no alert at all, only a database, and there the best model, by the authors' account, cleared their bar of 50% recall on 5 of 13 ATT&CK tactics.

What a 40% miss rate looks like in alerts

ALERT-BENCH reports rates, not counts. With the class counts from its Section III and the reported rates, the counts can be rebuilt, and for both systems below exactly one confusion matrix rounds to the published figures. The shipped script does this search and prints the result.

Same 1,247 alerts, GPT-4.1 Direct-Prompt (best studied design) AIDA (the authors' design)
Reported recall 0.596, F1 0.744 recall 0.969, precision 0.948
Attack alerts sent to an analyst 134 218
Attack alerts closed as benign 91 7
Benign alerts sent to an analyst 1 12
Benign alerts closed 1,021 1,010
Share of all alerts sent to people 10.8% 18.4%
Closed alerts that really were benign 91.8% 99.3%

Direct-Prompt bothered an analyst with exactly one benign alert in 1,247 and quietly closed 91 attack alerts. Measured by analyst workload alone, that looks like the better product, and the paper warns against reading it that way: "A system that dismissed every alert would report zero Investigation Load while missing every attack." AIDA sends 95 more alerts to people and misses 84 fewer attacks; its 18.4% load and 99.3% figure match the paper's own, which is the check that the rebuild is right. It also costs more, about $0.109 per alert against $0.016 to $0.107 for the studied designs, at the authors' GPT-4.1 prices.

Two cautions belong next to that table. AIDA was designed by the same authors after they studied the five designs failing on this benchmark, and it was scored on the same alerts. And the sample comes from one attack scenario, heavy on web reconnaissance, so the result says little yet about a credential-theft or insider case.

Why the misses happened

The paper's trace analysis is the useful part for anyone running a pilot, because it names behaviours you can log.

  • An empty search became a verdict. ReAct, a common agent design, dismissed 98.6% of attack alerts when none of its queries returned a record, against 30.3% when at least one did. Four of the five designs dismissed at least 88% of attack alerts in the no-records state. On benign alerts the same split moved dismissal by at most 2.8 points. A query that matches nothing says the query matched nothing, not that nothing happened.
  • Reviewing in the same context did harm. Self-Refine changed 95 decisions in its review step: 16 repaired an error and 79 broke a correct decision. Chain-of-Verification changed 157, repairing 25 and breaking 132.
  • More tokens did not buy more catches. Direct-Prompt used 5,075 tokens per alert and had the best F1; Self-Consistency used 28,002 and missed 31 more attacks.

AIDA's answer is procedural rather than clever. Escalation needs evidence of malicious activity; dismissal needs affirmative benign evidence or an investigation complete enough to justify closure. A challenger argues in a separate context, and a judge rules on the evidence. In the full run the judge reversed 31 proposed decisions, all of them benign alerts proposed for escalation, and all 31 reversals were right.

Retrieval decides log analysis too

The retrieval-augmented paper ran a baseline that many first pilots resemble: paste the raw logs into the prompt. Only 22 to 158 of the scenario's 3,694 log lines fit, 0.6% to 4.3%, and they were the earliest lines, mostly benign start-up activity. Every model identified the victim machine from those lines and missed all of the attack infrastructure that appeared later. With query-based filtering and retrieval, average recall over 17 malware-traffic scenarios was 94% for Claude Sonnet 4, 89% for DeepSeek V3 and 81% for a locally run Llama 3.1 70B; the authors put DeepSeek's cost at $0.008 per analysis, 15 times less than Claude's. If a summarisation pilot pastes the first N lines of a log, it is measuring the first N lines.

The log is written partly by the attacker

This is the finding that changes the design, not just the tuning. A security log records adversarial traffic, so some of its fields are chosen by the attacker: the user agent, the requested URL, the request body, a DNS name, the username tried at a login prompt. When a model reads those fields, it can read instructions planted in them.

The Watchtower paper tested that with gpt-4o-mini on synthetic logs across 48 combinations of attack style, defence and task. A plain "ignore your instructions" style line failed completely: 0% of malicious logs were relabelled. A line posing as a note from the security team got 68% of malicious logs labelled benign under naive prompting, and 15% with structured prompting. Summaries were the weakest task: one class of planted context steered 96% of summaries with no defence and 38% even with a constrained output format. Averaged over everything, injection success fell from 26.6% with naive prompting to 17.3% with structured prompting, 13.2% with field sanitisation and 11.8% with constrained output. The authors also built a deterministic simulator from earlier research, which predicted 92% success for the plain override that the real model ignored entirely. Simulated models are a poor stand-in for the model you deploy.

LogInject placed 2,569 adversarial entries among 10,278 benign ones and tested three models with no defences: success ran from 74.8% for Claude 3.5 Sonnet to 88.2% for Llama 3 70B, and the objective that worked best was hiding activity (89.2% on average). On GPT-4o the defences stacked: input filtering took success from 87.3% to 78.2%, marking the log as data with explicit delimiters and a trust instruction took it to 51.4%, and filtering, delimiting and output validation together took it to 8.4%, while accuracy on benign batches fell from 94.2% to 90.8%. The delimiter defence weakened as context grew, from 32% success at 1,000 tokens to 72% at 16,000. The paper also reports that instructions split across several log entries pass filters that check one entry at a time.

What the people in the SOC said

Two papers measured analysts rather than models. In interviews with 20 practitioners, people reported time saved on low-stakes work that is quick to verify, such as summarising logs or drafting first leads, and treated model output as a draft rather than a decision; verification was ad hoc rather than written procedure. In a field study at one real SOC, 12 employees triaged real alarms from their own environment: their decision was right in 83% of cases, but only 39% of their explanations named the actual root cause. People are not a perfect baseline either, and the explanation is the weak part on both sides, which is a reason to check a model's explanation against the evidence rather than against how convincing it reads.

What a defender takes from the record

  • Measure on your own alerts, scored against your analysts' final dispositions. Accuracy on a curated, balanced set does not carry over to a queue that is mostly benign.
  • Report the false-negative rate next to the workload figure, every time. A falling escalation rate is not a win until you know what it closed.
  • Never let "no records found" close an alert on its own. Log, per decision, how many records the investigation retrieved; the script below splits attack dismissals on exactly that.
  • Ask for evidence to dismiss, not only to escalate, and run any review step in a fresh context.
  • Retrieve, do not truncate. Filter and query the logs for the alert; do not paste whatever fits.
  • Treat attacker-written fields as untrusted input. Pass logs as structured fields, mark them as data, constrain the output to a schema, and keep a person on any closure that rests on a model's summary.
  • Re-test when the alert mix or the model changes. REFINE was built to hold recall at 1.0 and kept it on later windows in three of four scenarios; in the fourth it fell to 0.807.

What we could not check

  • arXiv had no HTML version of the Cyber Defense Benchmark paper (version 3), so its 3.8% and 5-of-13 figures are the abstract's; we did not read its tables. For REFINE, the interview study and the field study we read the abstracts only.
  • SIR-Bench scores the authors' own agent, does not name the model behind it, and had not released its test cases; the paper compares its agent with Tier-2 analysts using "SOC operational data" it does not cite, so that comparison is left out here.
  • LogInject's text says output validation combined with the delimiter defence brings success to 8.4%; its Table 9 gives 18.6% for that pair and 8.4% only for all three layers together. We used the table.
  • Both injection papers use synthetic or academic logs (LogHub and generated entries), not a production SOC's traffic, and each tests specific model versions that are already older than what vendors sell.
  • None of the ten papers measures a deployed system's miss rate over months in a live SOC, which is the number a buyer most needs.

The dataset

_README for llms-in-the-soc-alert-triage-what-papers-measured.csv, one row per reported figure, 45 rows:

  • paper, arxiv_id, version_read: the short name used in this article, the arXiv identifier, and the version read
  • task: alert triage, threat hunting, incident response, incident analysis, log injection, practice or human baseline
  • setting, data: what the system had to do, and what it was tested on, in the paper's terms
  • system, system_by_authors: the design measured, and yes where the paper's authors built it
  • model: the model named for that figure (none for studies of people)
  • metric, value, n: what was measured, the value as a fraction or count, and the sample size where the paper gives one
  • quote: the sentence fragment from the paper that states the figure
  • url, retrieved: the arXiv abstract page and the day it was read (2026-10-14)

Scoring your own pilot

The script has three modes and no network access. With no flags it prints the dataset grouped by task. With --rebuild it repeats the confusion-matrix search above:

def rebuild(pos, neg, recall, precision=None, f1=None):
    """Every (TP, FN, FP, TN) whose rates round to the reported ones (three decimals)."""
    tp = round(recall * pos)
    out = []
    for fp in range(neg + 1):
        m = metrics(tp, pos - tp, fp, neg - fp)
        if round(m["recall"], 3) != round(recall, 3):
            continue
        if precision is not None and round(m["precision"], 3) != round(precision, 3):
            continue
        if f1 is not None and round(m["f1"], 3) != round(f1, 3):
            continue
        out.append((tp, pos - tp, fp, neg - fp))
    return out

With --pilot it scores a CSV of your own, one row per alert with truth (attack or benign, from your analysts), decision (escalate or dismiss, from the model-assisted triage) and, if you log it, records_returned. It prints recall, the false-negative rate with a Wilson 95% interval, precision, the share of alerts sent to people, the share of closures that were really benign, and the attack dismissal rate with and without any returned records.

python code/llms-in-the-soc-alert-triage-what-papers-measured.py --rebuild
python code/llms-in-the-soc-alert-triage-what-papers-measured.py --pilot my_pilot.csv

If a pilot logs only one thing beyond the decision, make it records_returned. In the one benchmark that measured it, an agent whose searches came back empty closed 98.6% of the attack alerts in front of it.

Everything on this site is written for defenders and for people securing systems they own or are authorised to test. It describes how attacks work so they can be recognised and closed; it is not a guide to accessing anyone else's systems.

Sources

  1. Tsegai, Kantchelian, Yao and Gao, "From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage" (arXiv:2610.10608v1, 7 October 2026; abstract and full text read 2026-10-14)
  2. Chona, Kozlov and Kumar, "Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps" (arXiv:2604.19533v3, 23 April 2026; abstract read 2026-10-14)
  3. Jajodia, Sultana, Majumdar, Taylor and Vandenberghe, "Before You Hand Over the Wheel: Evaluating LLMs for Security Incident Analysis" (arXiv:2603.06422v1, 6 March 2026; full text read 2026-10-14)
  4. Begimher, Leo, Huang, Gaw and Zheng, "SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents" (arXiv:2604.12040v1, 13 April 2026; full text read 2026-10-14)
  5. Cadet et al., "Retrieval-Augmented LLMs for Security Incident Analysis" (arXiv:2603.18196v3, 4 May 2026; full text read 2026-10-14)
  6. Chen, Long and Wang, "REFINE: A Resilient Evolution Framework for Intelligent Enterprise Alert Triage in Security Operations Centers" (arXiv:2609.32516v1, 26 September 2026; abstract read 2026-10-14)
  7. Pandey and Bhujang, "Poisoning the Watchtower: Prompt Injection Attacks Against LLM-Augmented Security Operations Through Adversarial Log Content" (arXiv:2605.24421v1, 23 May 2026; full text read 2026-10-14)
  8. Karanjai, Lu, Madhavarao, Xu and Shi, "Context Contamination in LLM Analysis of Network Security Logs: Poison with Passive Prompt Injection and Mitigation Evaluation" (arXiv:2607.14493v1, 16 July 2026; full text read 2026-10-14)
  9. Rabieinejad, Dehghantanha, Zarrinkalam and Dastgerdy, "Beyond the Hype: Evaluating LLM Integration and Practical Limitations in Security Operation Centers" (arXiv:2608.17154v1, 17 August 2026; abstract read 2026-10-14)
  10. Moosmann, Pekaric and Apruzzese, "Can SOC Operators Explain their Decisions while Triaging Alarms? A Real-World Study" (arXiv:2604.22001v1, 23 April 2026; abstract read 2026-10-14)
  11. OpenAlex works API, used to find the papers (seven searches, publication date from 1 June 2024, run 2026-10-14)