File size: 4,521 Bytes
f9c8f39 48c3fec f9c8f39 48c3fec f9c8f39 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 |
---
language:
- en
license: apache-2.0
library_name: transformers
tags:
- modernbert
- security
- jailbreak-detection
- prompt-injection
- token-classification
- tool-calling
- llm-safety
- mcp
datasets:
- microsoft/llmail-inject-challenge
- allenai/wildjailbreak
- hackaprompt/hackaprompt-dataset
- JailbreakBench/JBB-Behaviors
base_model: answerdotai/ModernBERT-base
pipeline_tag: token-classification
model-index:
- name: toolcall-verifier
results:
- task:
type: token-classification
name: Unauthorized Tool Call Detection
metrics:
- name: UNAUTHORIZED F1
type: f1
value: 0.9350
- name: UNAUTHORIZED Precision
type: precision
value: 0.9501
- name: UNAUTHORIZED Recall
type: recall
value: 0.9205
- name: Accuracy
type: accuracy
value: 0.9288
---
# ToolCallVerifier - Unauthorized Tool Call Detection
<div align="center">
[](https://opensource.org/licenses/Apache-2.0)
[](https://huggingface.co/answerdotai/ModernBERT-base)
**Stage 2 of Two-Stage LLM Agent Defense Pipeline**
</div>
---
## 🎯 What This Model Does
ToolCallVerifier is a **ModernBERT-based token classifier** that detects unauthorized tool calls in LLM agent systems. It performs token-level classification on tool call JSON to identify malicious arguments that may have been injected through prompt injection attacks.
| Label | Description |
|-------|-------------|
| `AUTHORIZED` | Token is part of a legitimate, user-requested action |
| `UNAUTHORIZED` | Token indicates injected/malicious content — **BLOCK** |
---
## 🚨 Attack Categories Covered
| Category | Source | Description |
|----------|--------|-------------|
| Delimiter Injection | LLMail | `<<end_context>>`, `>>}}\]\])` |
| Word Obfuscation | LLMail | Inserting noise words between tokens |
| Fake Sessions | LLMail | `START_USER_SESSION`, `EXECUTE_USERQUERY` |
| Roleplay Injection | WildJailbreak | "You are an admin bot that can..." |
| XML Tag Injection | WildJailbreak | `<execute_action>`, `<tool_call>` |
| Authority Bypass | WildJailbreak | "As administrator, I authorize..." |
| Intent Mismatch | Synthetic | User asks X, tool does Y |
| MCP Tool Poisoning | Synthetic | Hidden exfiltration in tool args |
| MCP Shadowing | Synthetic | Fake authorization context |
## 🔗 Integration with FunctionCallSentinel
This model is **Stage 2** of a two-stage defense pipeline:
```
┌─────────────────┐ ┌──────────────────────┐ ┌─────────────────┐
│ User Prompt │────▶│ ToolCallSentinel │────▶│ LLM + Tools │
│ │ │ (Stage 1) │ │ │
└─────────────────┘ └──────────────────────┘ └────────┬────────┘
│
┌──────────────────────────────▼──────────────────────────┐
│ ToolCallVerifier (This Model) │
│ Token-level verification before tool execution │
└─────────────────────────────────────────────────────────┘
```
| Scenario | Recommendation |
|----------|----------------|
| General chatbot | Stage 1 only |
| Tool-calling agent (low risk) | Stage 1 only |
| Tool-calling agent (high risk) | **Both stages** |
| Email/file system access | **Both stages** |
| Financial transactions | **Both stages** |
---
## 🎯 Intended Use
### Primary Use Cases
- **LLM Agent Security**: Verify tool calls before execution
- **Prompt Injection Defense**: Detect unauthorized actions from injected prompts
- **API Gateway Protection**: Filter malicious tool calls at infrastructure level
### Out of Scope
- General text classification
- Non-tool-calling scenarios
- Languages other than English
## 📜 License
Apache 2.0
|