Shield Agents commited on
Commit ·
de31cf7
0
Parent(s):
🛡️ Initial release - Shield Agents v1.0.0
Browse filesMulti-agent AI cybersecurity scanner with 6 specialized agents:
- VulnAgent: SAST vulnerability detection
- ThreatAgent: Threat classification & MITRE mapping
- ReconAgent: Codebase reconnaissance
- ComplianceAgent: OWASP/PCI/HIPAA compliance checks
- ResponseAgent: Incident response playbooks
- AutoFixAgent: AI-powered auto-remediation
Features: SARIF output, .shieldignore, caching/incremental scans,
deduplication engine, LLM fallback, VS Code extension, CI mode,
JSON output, benchmark suite, and more.
This view is limited to 50 files because it contains too many changes. See raw diff
- .github/FUNDING.yml +4 -0
- .github/ISSUE_TEMPLATE/bug_report.md +30 -0
- .github/ISSUE_TEMPLATE/feature_request.md +19 -0
- .github/PULL_REQUEST_TEMPLATE.md +20 -0
- .gitignore +38 -0
- CHANGELOG.md +34 -0
- CONTRIBUTING.md +103 -0
- Dockerfile +24 -0
- LICENSE +21 -0
- README.md +473 -0
- benchmarks/__init__.py +1 -0
- benchmarks/benchmark.py +554 -0
- config.yaml +49 -0
- docker-compose.yml +25 -0
- examples/vulnerable_app.py +102 -0
- pyproject.toml +67 -0
- requirements-dev.txt +5 -0
- requirements.txt +2 -0
- setup.py +9 -0
- shield_agents/__init__.py +10 -0
- shield_agents/agents/__init__.py +19 -0
- shield_agents/agents/autofix.py +167 -0
- shield_agents/agents/base.py +90 -0
- shield_agents/agents/compliance.py +74 -0
- shield_agents/agents/recon.py +69 -0
- shield_agents/agents/response.py +67 -0
- shield_agents/agents/threat.py +69 -0
- shield_agents/agents/vuln.py +80 -0
- shield_agents/cache.py +238 -0
- shield_agents/cli.py +420 -0
- shield_agents/config.py +209 -0
- shield_agents/deduplication.py +254 -0
- shield_agents/knowledge/__init__.py +1 -0
- shield_agents/knowledge/cve.py +65 -0
- shield_agents/knowledge/mitre.py +61 -0
- shield_agents/knowledge/owasp.py +85 -0
- shield_agents/llm.py +794 -0
- shield_agents/orchestrator.py +382 -0
- shield_agents/report/__init__.py +6 -0
- shield_agents/report/generator.py +352 -0
- shield_agents/report/sarif.py +240 -0
- shield_agents/scanners/__init__.py +6 -0
- shield_agents/scanners/sast.py +319 -0
- shield_agents/scanners/secrets.py +324 -0
- shield_agents/shieldignore.py +298 -0
- shield_agents/utils/__init__.py +1 -0
- shield_agents/utils/crypto.py +49 -0
- shield_agents/utils/helpers.py +151 -0
- tests/__init__.py +1 -0
- tests/test_cache.py +99 -0
.github/FUNDING.yml
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# These funding options are just examples - customize as needed
|
| 2 |
+
github: shield-agents
|
| 3 |
+
patreon: # Replace with your Patreon username
|
| 4 |
+
open_collective: # Replace with your Open Collective slug
|
.github/ISSUE_TEMPLATE/bug_report.md
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
name: Bug report
|
| 3 |
+
about: Create a report to help us improve
|
| 4 |
+
title: '[BUG] '
|
| 5 |
+
labels: bug
|
| 6 |
+
assignees: ''
|
| 7 |
+
---
|
| 8 |
+
|
| 9 |
+
## Describe the Bug
|
| 10 |
+
A clear description of what the bug is.
|
| 11 |
+
|
| 12 |
+
## Steps to Reproduce
|
| 13 |
+
1. Run `shield-agents ...`
|
| 14 |
+
2. With config `...`
|
| 15 |
+
3. See error
|
| 16 |
+
|
| 17 |
+
## Expected Behavior
|
| 18 |
+
What you expected to happen.
|
| 19 |
+
|
| 20 |
+
## Actual Behavior
|
| 21 |
+
What actually happened.
|
| 22 |
+
|
| 23 |
+
## Environment
|
| 24 |
+
- OS: [e.g., Ubuntu 22.04]
|
| 25 |
+
- Python version: [e.g., 3.11]
|
| 26 |
+
- Shield Agents version: [e.g., 2.0.0]
|
| 27 |
+
- LLM Provider: [e.g., mock, openai]
|
| 28 |
+
|
| 29 |
+
## Additional Context
|
| 30 |
+
Any other context, logs, or screenshots.
|
.github/ISSUE_TEMPLATE/feature_request.md
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
name: Feature request
|
| 3 |
+
about: Suggest an idea for Shield Agents
|
| 4 |
+
title: '[FEATURE] '
|
| 5 |
+
labels: enhancement
|
| 6 |
+
assignees: ''
|
| 7 |
+
---
|
| 8 |
+
|
| 9 |
+
## Is your feature request related to a problem?
|
| 10 |
+
A clear description of what the problem is. Ex. I'm always frustrated when [...]
|
| 11 |
+
|
| 12 |
+
## Describe the Solution You'd Like
|
| 13 |
+
A clear description of what you want to happen.
|
| 14 |
+
|
| 15 |
+
## Describe Alternatives You've Considered
|
| 16 |
+
A clear description of any alternative solutions or features you've considered.
|
| 17 |
+
|
| 18 |
+
## Additional Context
|
| 19 |
+
Any other context, screenshots, or references.
|
.github/PULL_REQUEST_TEMPLATE.md
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
## Description
|
| 2 |
+
Brief description of changes.
|
| 3 |
+
|
| 4 |
+
## Type of Change
|
| 5 |
+
- [ ] Bug fix (non-breaking change that fixes an issue)
|
| 6 |
+
- [ ] New feature (non-breaking change that adds functionality)
|
| 7 |
+
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
|
| 8 |
+
- [ ] Documentation update
|
| 9 |
+
|
| 10 |
+
## Testing
|
| 11 |
+
- [ ] Unit tests pass (`pytest tests/ -v`)
|
| 12 |
+
- [ ] Benchmark suite passes (`python -m benchmarks.benchmark`)
|
| 13 |
+
- [ ] Manual testing performed
|
| 14 |
+
|
| 15 |
+
## Checklist
|
| 16 |
+
- [ ] Code follows project style guidelines
|
| 17 |
+
- [ ] Self-review performed
|
| 18 |
+
- [ ] Comments added for complex code
|
| 19 |
+
- [ ] Documentation updated (README, CHANGELOG)
|
| 20 |
+
- [ ] No new warnings introduced
|
.gitignore
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Python
|
| 2 |
+
__pycache__/
|
| 3 |
+
*.py[cod]
|
| 4 |
+
*$py.class
|
| 5 |
+
*.egg-info/
|
| 6 |
+
dist/
|
| 7 |
+
build/
|
| 8 |
+
*.egg
|
| 9 |
+
|
| 10 |
+
# Virtual environments
|
| 11 |
+
.venv/
|
| 12 |
+
venv/
|
| 13 |
+
env/
|
| 14 |
+
|
| 15 |
+
# IDE
|
| 16 |
+
.vscode/
|
| 17 |
+
.idea/
|
| 18 |
+
*.swp
|
| 19 |
+
*.swo
|
| 20 |
+
*~
|
| 21 |
+
|
| 22 |
+
# Shield Agents
|
| 23 |
+
.shield-cache/
|
| 24 |
+
shield-reports/
|
| 25 |
+
.shieldignore
|
| 26 |
+
|
| 27 |
+
# OS
|
| 28 |
+
.DS_Store
|
| 29 |
+
Thumbs.db
|
| 30 |
+
|
| 31 |
+
# Testing
|
| 32 |
+
.pytest_cache/
|
| 33 |
+
.coverage
|
| 34 |
+
htmlcov/
|
| 35 |
+
|
| 36 |
+
# Build
|
| 37 |
+
*.tar.gz
|
| 38 |
+
*.whl
|
CHANGELOG.md
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Changelog
|
| 2 |
+
|
| 3 |
+
All notable changes to Shield Agents will be documented in this file.
|
| 4 |
+
|
| 5 |
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
| 6 |
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
| 7 |
+
|
| 8 |
+
## [2.0.0] - 2025-05-29
|
| 9 |
+
|
| 10 |
+
### Added
|
| 11 |
+
- Smart Mock Provider with pattern-matched findings based on actual code content
|
| 12 |
+
- Deduplication Engine (3-phase: exact match, fuzzy match, category merge)
|
| 13 |
+
- SARIF 2.1.0 output for GitHub Security tab integration
|
| 14 |
+
- `.shieldignore` file support with 9 rule types
|
| 15 |
+
- Caching and incremental scanning
|
| 16 |
+
- LLM Fallback Handling with 5-strategy response parser
|
| 17 |
+
- VS Code Extension with on-save scanning and inline diagnostics
|
| 18 |
+
- Benchmark Suite with 13 OWASP WebGoat-style test cases
|
| 19 |
+
- Auto-Fix Agent (Master White Hat Hacker) with pattern-based and LLM fixes
|
| 20 |
+
- Agent-differentiated mock provider (each agent returns specialized findings)
|
| 21 |
+
- `--ci` mode for CI/CD pipelines (SARIF output, JSON to stdout, exit code based on risk)
|
| 22 |
+
- `--format json` for piping results to other tools
|
| 23 |
+
- Auto-exclude test/benchmark/examples directories from scan
|
| 24 |
+
- Migrated from setup.py to pyproject.toml
|
| 25 |
+
|
| 26 |
+
### Changed
|
| 27 |
+
- Mock provider now returns agent-specific findings instead of generic responses
|
| 28 |
+
- CLI now supports `--format` flag with rich/json/sarif/plain options
|
| 29 |
+
- Orchestrator auto-excludes content directories (tests, benchmarks, examples)
|
| 30 |
+
|
| 31 |
+
### Fixed
|
| 32 |
+
- `shield-agents init` crash when config directory doesn't exist
|
| 33 |
+
- JWT test assertion mismatch
|
| 34 |
+
- SAST missing patterns for path traversal and SSRF
|
CONTRIBUTING.md
ADDED
|
@@ -0,0 +1,103 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Contributing to Shield Agents
|
| 2 |
+
|
| 3 |
+
First off, thank you for considering contributing to Shield Agents! It's people like you that make this tool better for everyone.
|
| 4 |
+
|
| 5 |
+
## How to Contribute
|
| 6 |
+
|
| 7 |
+
### Reporting Bugs
|
| 8 |
+
|
| 9 |
+
Before creating bug reports, please check the existing issues. When you create a bug report, include as many details as possible:
|
| 10 |
+
|
| 11 |
+
- **OS and Python version**
|
| 12 |
+
- **Shield Agents version** (`shield-agents version`)
|
| 13 |
+
- **Steps to reproduce**
|
| 14 |
+
- **Expected vs actual behavior**
|
| 15 |
+
- **Scan output or error messages**
|
| 16 |
+
|
| 17 |
+
### Suggesting Enhancements
|
| 18 |
+
|
| 19 |
+
Enhancement suggestions are tracked as GitHub issues. Include:
|
| 20 |
+
|
| 21 |
+
- **Use case** - why is this needed?
|
| 22 |
+
- **Expected behavior** - what should it do?
|
| 23 |
+
- **Current workaround** - is there an alternative?
|
| 24 |
+
|
| 25 |
+
### Pull Requests
|
| 26 |
+
|
| 27 |
+
1. Fork the repository
|
| 28 |
+
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
|
| 29 |
+
3. Make your changes
|
| 30 |
+
4. Run the test suite:
|
| 31 |
+
```bash
|
| 32 |
+
pip install -e ".[dev]"
|
| 33 |
+
pytest tests/ -v
|
| 34 |
+
python -m benchmarks.benchmark
|
| 35 |
+
```
|
| 36 |
+
5. Ensure code style:
|
| 37 |
+
```bash
|
| 38 |
+
ruff check shield_agents/
|
| 39 |
+
black --check shield_agents/
|
| 40 |
+
```
|
| 41 |
+
6. Commit with a clear message (`git commit -m 'Add amazing feature'`)
|
| 42 |
+
7. Push to the branch (`git push origin feature/amazing-feature`)
|
| 43 |
+
8. Open a Pull Request
|
| 44 |
+
|
| 45 |
+
### Adding New SAST Rules
|
| 46 |
+
|
| 47 |
+
1. Add the rule to `shield_agents/scanners/sast.py` in the `SAST_RULES` list
|
| 48 |
+
2. Add a test case in `tests/test_sast.py`
|
| 49 |
+
3. Add a benchmark case in `benchmarks/benchmark.py` (optional but recommended)
|
| 50 |
+
4. Update the README with the new rule
|
| 51 |
+
|
| 52 |
+
### Adding New Secret Patterns
|
| 53 |
+
|
| 54 |
+
1. Add the pattern to `shield_agents/scanners/secrets.py` in `SECRET_PATTERNS`
|
| 55 |
+
2. Add a test case in `tests/test_secrets.py`
|
| 56 |
+
3. Consider entropy filtering to reduce false positives
|
| 57 |
+
|
| 58 |
+
### Adding New Agents
|
| 59 |
+
|
| 60 |
+
1. Create a new file in `shield_agents/agents/`
|
| 61 |
+
2. Extend `BaseAgent` and implement `analyze()` and `get_system_prompt()`
|
| 62 |
+
3. Add agent-specific patterns to the MockProvider in `llm.py`
|
| 63 |
+
4. Register in `shield_agents/agents/__init__.py`
|
| 64 |
+
5. Add configuration in `config.py`
|
| 65 |
+
6. Wire into the orchestrator in `orchestrator.py`
|
| 66 |
+
7. Add tests and update README
|
| 67 |
+
|
| 68 |
+
## Development Setup
|
| 69 |
+
|
| 70 |
+
```bash
|
| 71 |
+
# Clone the repository
|
| 72 |
+
git clone https://github.com/shield-agents/shield-agents.git
|
| 73 |
+
cd shield-agents
|
| 74 |
+
|
| 75 |
+
# Create a virtual environment
|
| 76 |
+
python -m venv venv
|
| 77 |
+
source venv/bin/activate # Linux/Mac
|
| 78 |
+
# or venv\Scripts\activate on Windows
|
| 79 |
+
|
| 80 |
+
# Install with dev dependencies
|
| 81 |
+
pip install -e ".[dev]"
|
| 82 |
+
|
| 83 |
+
# Run tests
|
| 84 |
+
pytest tests/ -v
|
| 85 |
+
|
| 86 |
+
# Run benchmarks
|
| 87 |
+
python -m benchmarks.benchmark --verbose
|
| 88 |
+
|
| 89 |
+
# Run linter
|
| 90 |
+
ruff check shield_agents/
|
| 91 |
+
```
|
| 92 |
+
|
| 93 |
+
## Code Style
|
| 94 |
+
|
| 95 |
+
- Follow PEP 8
|
| 96 |
+
- Use type hints
|
| 97 |
+
- Maximum line length: 100 characters
|
| 98 |
+
- Use `ruff` for linting
|
| 99 |
+
- Use `black` for formatting
|
| 100 |
+
|
| 101 |
+
## License
|
| 102 |
+
|
| 103 |
+
By contributing, you agree that your contributions will be licensed under the MIT License.
|
Dockerfile
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
FROM python:3.11-slim
|
| 2 |
+
|
| 3 |
+
LABEL maintainer="Shield Agents Contributors"
|
| 4 |
+
LABEL description="AI-Powered Multi-Agent Cybersecurity Scanner"
|
| 5 |
+
LABEL version="2.0.0"
|
| 6 |
+
|
| 7 |
+
WORKDIR /app
|
| 8 |
+
|
| 9 |
+
# Install dependencies
|
| 10 |
+
COPY requirements.txt .
|
| 11 |
+
RUN pip install --no-cache-dir -r requirements.txt
|
| 12 |
+
|
| 13 |
+
# Copy package
|
| 14 |
+
COPY . .
|
| 15 |
+
|
| 16 |
+
# Install the package
|
| 17 |
+
RUN pip install --no-cache-dir -e .
|
| 18 |
+
|
| 19 |
+
# Create reports directory
|
| 20 |
+
RUN mkdir -p /app/shield-reports
|
| 21 |
+
|
| 22 |
+
# Default command
|
| 23 |
+
ENTRYPOINT ["shield-agents"]
|
| 24 |
+
CMD ["scan", "--help"]
|
LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2024 Shield Agents Contributors
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
README.md
ADDED
|
@@ -0,0 +1,473 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<div align="center">
|
| 2 |
+
|
| 3 |
+
# 🛡️ Shield Agents
|
| 4 |
+
|
| 5 |
+
### AI-Powered Multi-Agent Cybersecurity Scanner
|
| 6 |
+
|
| 7 |
+
[](https://www.python.org/downloads/)
|
| 8 |
+
[](https://opensource.org/licenses/MIT)
|
| 9 |
+
[](https://github.com/astral-sh/ruff)
|
| 10 |
+
[]()
|
| 11 |
+
[]()
|
| 12 |
+
|
| 13 |
+
**Production-grade security analysis using coordinated AI agents.**
|
| 14 |
+
Detect vulnerabilities, threats, secrets, and compliance issues — then auto-fix them.
|
| 15 |
+
|
| 16 |
+
[Getting Started](#-getting-started) · [Features](#-features) · [Architecture](#-architecture) · [CI/CD](#-cicd-integration) · [VS Code](#-vs-code-extension) · [Contributing](CONTRIBUTING.md)
|
| 17 |
+
|
| 18 |
+
</div>
|
| 19 |
+
|
| 20 |
+
---
|
| 21 |
+
|
| 22 |
+
## ✨ Why Shield Agents?
|
| 23 |
+
|
| 24 |
+
Most security scanners just find problems. Shield Agents **finds AND fixes them** using a team of specialized AI agents that work together like a real security team:
|
| 25 |
+
|
| 26 |
+
- 🔍 **VulnAgent** — Finds SQL injection, XSS, command injection, and more
|
| 27 |
+
- 🎯 **ThreatAgent** — Maps attack vectors to MITRE ATT&CK techniques
|
| 28 |
+
- 🕵️ **ReconAgent** — Detects information disclosure and exposed secrets
|
| 29 |
+
- 📋 **ComplianceAgent** — Checks against OWASP Top 10 2021
|
| 30 |
+
- 🚨 **ResponseAgent** — Provides risk assessment and incident response plans
|
| 31 |
+
- 🔧 **AutoFixAgent** — The "Master White Hat Hacker" that generates copy-paste-ready code fixes
|
| 32 |
+
|
| 33 |
+
Works out of the box with the **smart mock provider** — no API key needed to start scanning.
|
| 34 |
+
|
| 35 |
+
---
|
| 36 |
+
|
| 37 |
+
## 🚀 Getting Started
|
| 38 |
+
|
| 39 |
+
### Installation
|
| 40 |
+
|
| 41 |
+
```bash
|
| 42 |
+
# Install from source (mock provider included - no API key needed)
|
| 43 |
+
pip install -e .
|
| 44 |
+
|
| 45 |
+
# With OpenAI support
|
| 46 |
+
pip install -e ".[openai]"
|
| 47 |
+
|
| 48 |
+
# With all LLM providers (OpenAI + Anthropic + Ollama)
|
| 49 |
+
pip install -e ".[all]"
|
| 50 |
+
|
| 51 |
+
# With development tools
|
| 52 |
+
pip install -e ".[dev]"
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
### Your First Scan
|
| 56 |
+
|
| 57 |
+
```bash
|
| 58 |
+
# Scan a project (uses smart mock provider by default)
|
| 59 |
+
shield-agents scan ./my-project
|
| 60 |
+
|
| 61 |
+
# Scan with auto-fix suggestions
|
| 62 |
+
shield-agents scan ./my-project --fix
|
| 63 |
+
|
| 64 |
+
# Full scan (ignore cache, scan everything)
|
| 65 |
+
shield-agents scan ./my-project --full
|
| 66 |
+
|
| 67 |
+
# Initialize configuration files
|
| 68 |
+
shield-agents init
|
| 69 |
+
```
|
| 70 |
+
|
| 71 |
+
### Docker
|
| 72 |
+
|
| 73 |
+
```bash
|
| 74 |
+
# Build and scan
|
| 75 |
+
docker-compose up shield-agents
|
| 76 |
+
|
| 77 |
+
# With Ollama for local LLM
|
| 78 |
+
docker-compose up ollama shield-agents
|
| 79 |
+
```
|
| 80 |
+
|
| 81 |
+
---
|
| 82 |
+
|
| 83 |
+
## 🛡️ Features
|
| 84 |
+
|
| 85 |
+
### Core Scanning Engine
|
| 86 |
+
|
| 87 |
+
| Feature | Details |
|
| 88 |
+
|---------|---------|
|
| 89 |
+
| **SAST Scanner** | 10 detection rules: SQL Injection, XSS, Command Injection, Path Traversal, Insecure Deserialization, Weak Crypto, Auth Issues, Hardcoded Credentials, SSL/TLS Issues, SSRF |
|
| 90 |
+
| **Secrets Scanner** | 24 pattern types: AWS, GitHub, Google, Slack, Stripe, DB connections, JWTs, Private Keys, and generic secrets with Shannon entropy filtering |
|
| 91 |
+
| **6 AI Agents** | VulnAgent, ThreatAgent, ReconAgent, ComplianceAgent, ResponseAgent, AutoFixAgent |
|
| 92 |
+
| **Auto-Fix Engine** | Pattern-based instant fixes + LLM-powered deep remediation with copy-paste-ready code |
|
| 93 |
+
|
| 94 |
+
### Production Features (v2.0)
|
| 95 |
+
|
| 96 |
+
| # | Feature | Description |
|
| 97 |
+
|---|---------|-------------|
|
| 98 |
+
| 1 | **Smart Mock Provider** | Pattern-matched findings based on actual code content — demo mode feels real, not static |
|
| 99 |
+
| 2 | **Deduplication Engine** | 3-phase dedup (exact → fuzzy → category merge) — when VulnAgent and SAST both find "SQL Injection", they merge into one finding with multiple sources |
|
| 100 |
+
| 3 | **SARIF 2.1.0 Output** | GitHub Security tab integration — upload results directly to GitHub code scanning |
|
| 101 |
+
| 4 | **`.shieldignore` File** | Like `.gitignore` for false positives — 9 rule types (file, category, severity, rule, path, line, title, cwe, id) |
|
| 102 |
+
| 5 | **Caching / Incremental Scans** | Cache previous results, only re-scan changed files — production-ready performance |
|
| 103 |
+
| 6 | **LLM Fallback Parser** | 5-strategy parser for when LLMs return invalid JSON (~30% failure rate) — never crash on bad responses |
|
| 104 |
+
| 7 | **VS Code Extension** | On-save scanning with inline diagnostics — security issues appear directly in your editor |
|
| 105 |
+
| 8 | **Benchmark Suite** | 13 OWASP WebGoat-style test cases — proves the scanner actually finds real bugs |
|
| 106 |
+
| 9 | **CI/CD Mode** | `--ci` flag for pipelines — JSON to stdout, SARIF output, exit code based on risk threshold |
|
| 107 |
+
| 10 | **`--format json`** | Pipe results to other tools — structured JSON output for integration |
|
| 108 |
+
| 11 | **Auto-Exclude** | Tests, benchmarks, and examples automatically excluded from scan — eliminates ~70% false positives |
|
| 109 |
+
| 12 | **Agent-Differentiated Mock** | Each AI agent returns specialized findings in mock mode — VulnAgent finds vulns, ThreatAgent finds threats |
|
| 110 |
+
|
| 111 |
+
---
|
| 112 |
+
|
| 113 |
+
## 📋 CLI Reference
|
| 114 |
+
|
| 115 |
+
```bash
|
| 116 |
+
shield-agents scan <target> [options]
|
| 117 |
+
|
| 118 |
+
Options:
|
| 119 |
+
--config, -c Path to config YAML file
|
| 120 |
+
--full Full scan (ignore cache)
|
| 121 |
+
--fix Generate auto-fix suggestions
|
| 122 |
+
--no-report Skip report generation
|
| 123 |
+
--sarif-only Output only SARIF format
|
| 124 |
+
--output, -o Output directory for reports
|
| 125 |
+
--provider LLM provider: mock, openai, anthropic, ollama
|
| 126 |
+
--format, -f Output format: rich, json, sarif, plain (default: rich)
|
| 127 |
+
--ci CI/CD mode: silent except JSON on stdout
|
| 128 |
+
--fail-threshold Risk score threshold for CI failure (default: 75)
|
| 129 |
+
--no-cache Disable caching
|
| 130 |
+
--no-dedup Disable deduplication
|
| 131 |
+
--no-ignore Ignore .shieldignore rules
|
| 132 |
+
--verbose, -v Verbose output
|
| 133 |
+
--debug Debug output
|
| 134 |
+
|
| 135 |
+
Commands:
|
| 136 |
+
scan <target> Run security scan
|
| 137 |
+
init Create .shieldignore and config templates
|
| 138 |
+
cache --stats Show cache statistics
|
| 139 |
+
cache --clear Clear scan cache
|
| 140 |
+
version Show version info
|
| 141 |
+
```
|
| 142 |
+
|
| 143 |
+
---
|
| 144 |
+
|
| 145 |
+
## ⚙️ Configuration
|
| 146 |
+
|
| 147 |
+
### Environment Variables
|
| 148 |
+
|
| 149 |
+
| Variable | Description | Default |
|
| 150 |
+
|----------|-------------|---------|
|
| 151 |
+
| `SHIELD_LLM_PROVIDER` | LLM provider (mock, openai, anthropic, ollama) | `mock` |
|
| 152 |
+
| `SHIELD_LLM_API_KEY` | API key for the LLM provider | None |
|
| 153 |
+
| `SHIELD_LLM_MODEL` | Model name | `gpt-4` |
|
| 154 |
+
| `SHIELD_LLM_BASE_URL` | Custom API base URL (for Ollama) | None |
|
| 155 |
+
| `SHIELD_VERBOSE` | Enable verbose output | `false` |
|
| 156 |
+
| `SHIELD_DEBUG` | Enable debug output | `false` |
|
| 157 |
+
|
| 158 |
+
### Config File (`config.yaml`)
|
| 159 |
+
|
| 160 |
+
```yaml
|
| 161 |
+
llm:
|
| 162 |
+
provider: mock # mock, openai, anthropic, ollama
|
| 163 |
+
model: gpt-4
|
| 164 |
+
temperature: 0.1
|
| 165 |
+
fallback_enabled: true # Robust JSON fallback parser
|
| 166 |
+
|
| 167 |
+
scanner:
|
| 168 |
+
sast_enabled: true
|
| 169 |
+
secrets_enabled: true
|
| 170 |
+
|
| 171 |
+
cache:
|
| 172 |
+
enabled: true
|
| 173 |
+
incremental: true # Only re-scan changed files
|
| 174 |
+
|
| 175 |
+
deduplication:
|
| 176 |
+
enabled: true
|
| 177 |
+
merge_sources: true # Merge duplicate findings from different sources
|
| 178 |
+
|
| 179 |
+
report:
|
| 180 |
+
output_dir: ./shield-reports
|
| 181 |
+
formats:
|
| 182 |
+
- html
|
| 183 |
+
- sarif
|
| 184 |
+
- json
|
| 185 |
+
```
|
| 186 |
+
|
| 187 |
+
### `.shieldignore` File
|
| 188 |
+
|
| 189 |
+
Suppress false positives with 9 rule types:
|
| 190 |
+
|
| 191 |
+
```gitignore
|
| 192 |
+
# Shield Agents Ignore File
|
| 193 |
+
category:information-disclosure # Ignore all info-disclosure findings
|
| 194 |
+
severity:LOW # Ignore LOW and INFO findings
|
| 195 |
+
rule:SAST-001 # Ignore a specific rule
|
| 196 |
+
file:test_app.py # Ignore all findings in a file
|
| 197 |
+
path:*test* # Ignore findings in test files
|
| 198 |
+
line:42:app.py # Ignore finding at specific line
|
| 199 |
+
title:Assertion* # Ignore findings with matching title (glob)
|
| 200 |
+
cwe:CWE-617 # Ignore findings with matching CWE
|
| 201 |
+
id:VulnAgent-3 # Ignore a specific finding by ID
|
| 202 |
+
```
|
| 203 |
+
|
| 204 |
+
---
|
| 205 |
+
|
| 206 |
+
## 🏗️ Architecture
|
| 207 |
+
|
| 208 |
+
```
|
| 209 |
+
shield-agents/
|
| 210 |
+
├── shield_agents/
|
| 211 |
+
│ ├── agents/ # 6 AI agents
|
| 212 |
+
│ │ ├── base.py # Abstract base agent
|
| 213 |
+
│ │ ├── vuln.py # Vulnerability detection
|
| 214 |
+
│ │ ├── threat.py # Threat modeling & MITRE ATT&CK
|
| 215 |
+
│ │ ├── recon.py # Reconnaissance & info disclosure
|
| 216 |
+
│ │ ├── compliance.py # OWASP compliance checking
|
| 217 |
+
│ │ ├── response.py # Incident response & risk assessment
|
| 218 |
+
│ │ └── autofix.py # Auto-remediation (Master White Hat Hacker)
|
| 219 |
+
│ ├── scanners/ # Static analysis
|
| 220 |
+
│ │ ├── sast.py # 10 SAST detection rules
|
| 221 |
+
│ │ └── secrets.py # 24 secret type patterns
|
| 222 |
+
│ ├── report/ # Output generation
|
| 223 |
+
│ │ ├── generator.py # HTML + JSON reports
|
| 224 |
+
│ │ └── sarif.py # SARIF 2.1.0 for GitHub
|
| 225 |
+
│ ├── knowledge/ # Security knowledge bases
|
| 226 |
+
│ │ ├── owasp.py # OWASP Top 10 2021
|
| 227 |
+
│ │ ├── mitre.py # MITRE ATT&CK
|
| 228 |
+
│ │ └── cve.py # Known CVE database
|
| 229 |
+
│ ├── utils/ # Utilities
|
| 230 |
+
│ │ ├── crypto.py # Hashing for cache
|
| 231 |
+
│ │ └── helpers.py # File I/O, risk scoring
|
| 232 |
+
│ ├── config.py # Configuration management
|
| 233 |
+
│ ├── llm.py # LLM providers + 5-strategy fallback parser
|
| 234 |
+
│ ├── deduplication.py # 3-phase finding deduplication engine
|
| 235 |
+
│ ├── cache.py # Incremental scan caching
|
| 236 |
+
│ ├── shieldignore.py # .shieldignore file parser (9 rule types)
|
| 237 |
+
│ ├── orchestrator.py # Central coordinator (6-phase pipeline)
|
| 238 |
+
│ └── cli.py # Rich-powered CLI
|
| 239 |
+
├── vscode-extension/ # VS Code extension
|
| 240 |
+
├── benchmarks/ # OWASP-style benchmark suite (13 cases)
|
| 241 |
+
├��─ tests/ # 52 unit tests
|
| 242 |
+
├── examples/ # Vulnerable example app
|
| 243 |
+
├── .github/ # Issue/PR templates
|
| 244 |
+
├── CHANGELOG.md # Version history
|
| 245 |
+
├── CONTRIBUTING.md # Contribution guide
|
| 246 |
+
└── pyproject.toml # Modern Python packaging
|
| 247 |
+
```
|
| 248 |
+
|
| 249 |
+
### 6-Phase Scan Pipeline
|
| 250 |
+
|
| 251 |
+
```
|
| 252 |
+
┌─────────────┐ ┌──────────────────┐ ┌───────────────┐
|
| 253 |
+
│ Phase 1 │ │ Phase 2 │ │ Phase 3 │
|
| 254 |
+
│ SAST Scan │──▶│ AI Agent Analysis│──▶│ Deduplication │
|
| 255 |
+
│ + Secrets │ │ (4 agents/file) │ │ (3 phases) │
|
| 256 |
+
└─────────────┘ └──────────────────┘ └───────────────┘
|
| 257 |
+
│
|
| 258 |
+
┌─────────────┐ ┌──────────────────┐ ┌───────────────┐
|
| 259 |
+
│ Phase 6 │ │ Phase 5 │ │ Phase 4 │
|
| 260 |
+
│ Reporting │◀──│ Auto-Fix Gen │◀──│ .shieldignore │
|
| 261 |
+
│ HTML/SARIF │ │ (Pattern + LLM) │ │ Filtering │
|
| 262 |
+
└─────────────┘ └──────────────────┘ └───────────────┘
|
| 263 |
+
```
|
| 264 |
+
|
| 265 |
+
---
|
| 266 |
+
|
| 267 |
+
## 🔄 CI/CD Integration
|
| 268 |
+
|
| 269 |
+
### GitHub Actions
|
| 270 |
+
|
| 271 |
+
```yaml
|
| 272 |
+
name: Security Scan
|
| 273 |
+
on: [push, pull_request]
|
| 274 |
+
|
| 275 |
+
jobs:
|
| 276 |
+
security:
|
| 277 |
+
runs-on: ubuntu-latest
|
| 278 |
+
steps:
|
| 279 |
+
- uses: actions/checkout@v4
|
| 280 |
+
- uses: actions/setup-python@v5
|
| 281 |
+
with:
|
| 282 |
+
python-version: '3.11'
|
| 283 |
+
- name: Install Shield Agents
|
| 284 |
+
run: pip install -e .
|
| 285 |
+
- name: Run Security Scan
|
| 286 |
+
run: shield-agents scan ./src --ci --fail-threshold 75
|
| 287 |
+
- name: Upload SARIF
|
| 288 |
+
if: always()
|
| 289 |
+
uses: github/codeql-action/upload-sarif@v3
|
| 290 |
+
with:
|
| 291 |
+
sarif_file: shield-reports/results.sarif
|
| 292 |
+
```
|
| 293 |
+
|
| 294 |
+
### GitLab CI
|
| 295 |
+
|
| 296 |
+
```yaml
|
| 297 |
+
security-scan:
|
| 298 |
+
stage: test
|
| 299 |
+
image: python:3.11-slim
|
| 300 |
+
script:
|
| 301 |
+
- pip install -e .
|
| 302 |
+
- shield-agents scan ./src --ci --format json > scan-results.json
|
| 303 |
+
artifacts:
|
| 304 |
+
paths:
|
| 305 |
+
- shield-reports/
|
| 306 |
+
reports:
|
| 307 |
+
sast: shield-reports/results.sarif
|
| 308 |
+
```
|
| 309 |
+
|
| 310 |
+
### Generic Pipeline (JSON to stdout)
|
| 311 |
+
|
| 312 |
+
```bash
|
| 313 |
+
# JSON output for any CI system
|
| 314 |
+
shield-agents scan ./src --ci --format json
|
| 315 |
+
|
| 316 |
+
# Exit code: 0 = pass, 1 = risk score exceeds threshold
|
| 317 |
+
shield-agents scan ./src --ci --fail-threshold 60
|
| 318 |
+
```
|
| 319 |
+
|
| 320 |
+
---
|
| 321 |
+
|
| 322 |
+
## 💻 VS Code Extension
|
| 323 |
+
|
| 324 |
+
Real-time security scanning directly in your editor:
|
| 325 |
+
|
| 326 |
+
- **Scan on Save** — Automatically scans files when you save
|
| 327 |
+
- **Inline Diagnostics** — Security findings appear as editor warnings/errors
|
| 328 |
+
- **Severity Highlighting** — Color-coded CRITICAL/HIGH/MEDIUM/LOW indicators
|
| 329 |
+
- **Quick Fix Suggestions** — Remediation advice for each finding
|
| 330 |
+
- **Workspace Scan** — Scan your entire project with one command
|
| 331 |
+
|
| 332 |
+
Install the extension from the `vscode-extension/` directory and configure via VS Code settings.
|
| 333 |
+
|
| 334 |
+
---
|
| 335 |
+
|
| 336 |
+
## 🤖 AI Assistant Compatibility
|
| 337 |
+
|
| 338 |
+
Shield Agents works seamlessly with AI coding assistants:
|
| 339 |
+
|
| 340 |
+
| Tool | How to Use |
|
| 341 |
+
|------|-----------|
|
| 342 |
+
| **Cursor** | Add `shield-agents scan` to `.cursorrules` for auto-scanning on changes |
|
| 343 |
+
| **Claude Code** | Run `shield-agents scan` in terminal, integrate findings into workflow |
|
| 344 |
+
| **GitHub Copilot** | VS Code extension provides inline security diagnostics alongside Copilot |
|
| 345 |
+
| **Aider** | Use `--fix` mode and pipe auto-fix suggestions for code modifications |
|
| 346 |
+
| **Windsurf** | CLI-based integration via terminal |
|
| 347 |
+
|
| 348 |
+
---
|
| 349 |
+
|
| 350 |
+
## 📊 Benchmark Suite
|
| 351 |
+
|
| 352 |
+
Verify detection accuracy with 13 OWASP WebGoat-style test cases:
|
| 353 |
+
|
| 354 |
+
```bash
|
| 355 |
+
python -m benchmarks.benchmark --verbose
|
| 356 |
+
```
|
| 357 |
+
|
| 358 |
+
| Category | Test Cases |
|
| 359 |
+
|----------|-----------|
|
| 360 |
+
| SQL Injection | String concat, format strings |
|
| 361 |
+
| XSS | DOM-based, template injection |
|
| 362 |
+
| Command Injection | os.system, subprocess |
|
| 363 |
+
| Path Traversal | User-controlled file paths |
|
| 364 |
+
| Insecure Deserialization | pickle, yaml, marshal |
|
| 365 |
+
| Weak Cryptography | MD5, SHA-1, random module |
|
| 366 |
+
| Hardcoded Secrets | Passwords, API keys, AWS credentials |
|
| 367 |
+
| SSL/TLS Issues | verify=False, unverified context |
|
| 368 |
+
| SSRF | User-controlled URLs |
|
| 369 |
+
| Auth Bypass | Assertion-based, session manipulation |
|
| 370 |
+
| Clean Code | Negative test (minimal findings) |
|
| 371 |
+
|
| 372 |
+
---
|
| 373 |
+
|
| 374 |
+
## 🧪 Running Tests
|
| 375 |
+
|
| 376 |
+
```bash
|
| 377 |
+
# Run all 52 unit tests
|
| 378 |
+
pytest tests/ -v
|
| 379 |
+
|
| 380 |
+
# Run specific test module
|
| 381 |
+
pytest tests/test_sast.py -v
|
| 382 |
+
pytest tests/test_secrets.py -v
|
| 383 |
+
pytest tests/test_llm.py -v
|
| 384 |
+
|
| 385 |
+
# Run benchmarks
|
| 386 |
+
python -m benchmarks.benchmark --verbose
|
| 387 |
+
|
| 388 |
+
# Lint
|
| 389 |
+
ruff check shield_agents/
|
| 390 |
+
```
|
| 391 |
+
|
| 392 |
+
---
|
| 393 |
+
|
| 394 |
+
## 🤝 Contributing
|
| 395 |
+
|
| 396 |
+
We love contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.
|
| 397 |
+
|
| 398 |
+
Quick start:
|
| 399 |
+
|
| 400 |
+
```bash
|
| 401 |
+
git clone https://github.com/shield-agents/shield-agents.git
|
| 402 |
+
cd shield-agents
|
| 403 |
+
pip install -e ".[dev]"
|
| 404 |
+
pytest tests/ -v
|
| 405 |
+
```
|
| 406 |
+
|
| 407 |
+
---
|
| 408 |
+
|
| 409 |
+
## 📄 License
|
| 410 |
+
|
| 411 |
+
MIT License — see [LICENSE](LICENSE) for details.
|
| 412 |
+
|
| 413 |
+
---
|
| 414 |
+
|
| 415 |
+
<div align="center">
|
| 416 |
+
|
| 417 |
+
### 🌐 En Español
|
| 418 |
+
|
| 419 |
+
**Shield Agents — Escáner de Ciberseguridad Multi-Agente con IA**
|
| 420 |
+
|
| 421 |
+
Shield Agents es una plataforma de análisis de seguridad de grado producción que utiliza agentes de IA coordinados para detectar vulnerabilidades, amenazas, secretos y problemas de cumplimiento en tu código fuente. No solo encuentra los problemas — los **arregla automáticamente**.
|
| 422 |
+
|
| 423 |
+
**Características principales:**
|
| 424 |
+
|
| 425 |
+
| Característica | Descripción |
|
| 426 |
+
|---------------|-------------|
|
| 427 |
+
| 🛡️ Escáner SAST | 10 reglas de detección: Inyección SQL, XSS, Inyección de Comandos, Path Traversal, Deserialización Insegura, Criptografía Débil, Problemas de Autenticación, Credenciales Hardcodeadas, SSL/TLS, SSRF |
|
| 428 |
+
| 🔑 Escáner de Secretos | 24 tipos de patrones: AWS, GitHub, Google, Slack, Stripe, conexiones DB, JWTs, Claves Privadas, con filtrado de entropía Shannon |
|
| 429 |
+
| 🤖 6 Agentes de IA | VulnAgent (vulnerabilidades), ThreatAgent (modelado de amenazas), ReconAgent (reconocimiento), ComplianceAgent (cumplimiento OWASP), ResponseAgent (respuesta a incidentes), AutoFixAgent (corrección automática — el "Hacker Ético Maestro") |
|
| 430 |
+
| 🔄 Motor de Deduplicación | 3 fases: coincidencia exacta → coincidencia difusa → fusión por categoría |
|
| 431 |
+
| 📋 Formato SARIF | Integración con la pestaña de Seguridad de GitHub |
|
| 432 |
+
| 🚫 Archivo `.shieldignore` | Como `.gitignore` pero para falsos positivos — 9 tipos de reglas |
|
| 433 |
+
| ⚡ Escaneo Incremental | Caché de resultados anteriores, solo re-escanea archivos modificados |
|
| 434 |
+
| 🔧 Auto-Fix | Correcciones instantáneas basadas en patrones + correcciones profundas con LLM |
|
| 435 |
+
| 💻 Extensión VS Code | Escaneo al guardar con diagnósticos inline |
|
| 436 |
+
| 🚀 Modo CI/CD | `--ci` para pipelines — JSON a stdout, SARIF, código de salida basado en riesgo |
|
| 437 |
+
|
| 438 |
+
**Inicio rápido:**
|
| 439 |
+
|
| 440 |
+
```bash
|
| 441 |
+
# Instalar (proveedor mock incluido — sin API key necesario)
|
| 442 |
+
pip install -e .
|
| 443 |
+
|
| 444 |
+
# Escanear un proyecto
|
| 445 |
+
shield-agents scan ./mi-proyecto
|
| 446 |
+
|
| 447 |
+
# Escanear con sugerencias de corrección automática
|
| 448 |
+
shield-agents scan ./mi-proyecto --fix
|
| 449 |
+
|
| 450 |
+
# Modo CI/CD
|
| 451 |
+
shield-agents scan ./src --ci --fail-threshold 75
|
| 452 |
+
|
| 453 |
+
# Inicializar configuración
|
| 454 |
+
shield-agents init
|
| 455 |
+
```
|
| 456 |
+
|
| 457 |
+
**Compatibilidad con asistentes de IA:**
|
| 458 |
+
|
| 459 |
+
Shield Agents es compatible con Cursor, Claude Code, GitHub Copilot, Aider y Windsurf. Funciona como una herramienta CLI estándar que se integra en cualquier flujo de trabajo de desarrollo.
|
| 460 |
+
|
| 461 |
+
**Contribuir:**
|
| 462 |
+
|
| 463 |
+
Las contribuciones son bienvenidas. Lee [CONTRIBUTING.md](CONTRIBUTING.md) para las guías detalladas. Los reportes de bugs, solicitudes de funcionalidades y pull requests son apreciados.
|
| 464 |
+
|
| 465 |
+
**Licencia:** MIT
|
| 466 |
+
|
| 467 |
+
---
|
| 468 |
+
|
| 469 |
+
If you find Shield Agents useful, please ⭐ star the repo — it helps others discover the project!
|
| 470 |
+
|
| 471 |
+
**Built with care for the security community**
|
| 472 |
+
|
| 473 |
+
</div>
|
benchmarks/__init__.py
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
"""Benchmark suite for Shield Agents."""
|
benchmarks/benchmark.py
ADDED
|
@@ -0,0 +1,554 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""
|
| 3 |
+
Benchmark Suite for Shield Agents.
|
| 4 |
+
|
| 5 |
+
Tests the scanner against known vulnerable code samples to verify
|
| 6 |
+
detection accuracy. Uses OWASP WebGoat-inspired test cases and
|
| 7 |
+
other standard vulnerability benchmarks.
|
| 8 |
+
|
| 9 |
+
Usage:
|
| 10 |
+
python -m benchmarks.benchmark
|
| 11 |
+
python -m benchmarks.benchmark --verbose
|
| 12 |
+
python -m benchmarks.benchmark --category injection
|
| 13 |
+
"""
|
| 14 |
+
|
| 15 |
+
import asyncio
|
| 16 |
+
import json
|
| 17 |
+
import logging
|
| 18 |
+
import os
|
| 19 |
+
import sys
|
| 20 |
+
import time
|
| 21 |
+
from dataclasses import dataclass, field
|
| 22 |
+
from pathlib import Path
|
| 23 |
+
from typing import Any, Dict, List, Optional, Tuple
|
| 24 |
+
|
| 25 |
+
# Add parent directory to path for imports
|
| 26 |
+
sys.path.insert(0, str(Path(__file__).parent.parent))
|
| 27 |
+
|
| 28 |
+
from shield_agents.config import ShieldConfig
|
| 29 |
+
from shield_agents.orchestrator import Orchestrator
|
| 30 |
+
from shield_agents.scanners import SASTScanner, SecretsScanner
|
| 31 |
+
|
| 32 |
+
logger = logging.getLogger("shield_agents.benchmark")
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
@dataclass
|
| 36 |
+
class BenchmarkCase:
|
| 37 |
+
"""A single benchmark test case."""
|
| 38 |
+
name: str
|
| 39 |
+
category: str
|
| 40 |
+
code: str
|
| 41 |
+
language: str
|
| 42 |
+
expected_findings: List[str] # Expected vulnerability categories
|
| 43 |
+
expected_min_severity: str = "MEDIUM"
|
| 44 |
+
description: str = ""
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
@dataclass
|
| 48 |
+
class BenchmarkResult:
|
| 49 |
+
"""Result of running a benchmark case."""
|
| 50 |
+
name: str
|
| 51 |
+
category: str
|
| 52 |
+
passed: bool
|
| 53 |
+
detected_categories: List[str]
|
| 54 |
+
expected_categories: List[str]
|
| 55 |
+
missed_categories: List[str]
|
| 56 |
+
false_positives: List[str]
|
| 57 |
+
findings_count: int
|
| 58 |
+
duration: float
|
| 59 |
+
details: str = ""
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
# =============================================================================
|
| 63 |
+
# Benchmark Test Cases - Based on OWASP WebGoat and real-world vulnerabilities
|
| 64 |
+
# =============================================================================
|
| 65 |
+
|
| 66 |
+
BENCHMARK_CASES: List[BenchmarkCase] = [
|
| 67 |
+
# --- SQL Injection ---
|
| 68 |
+
BenchmarkCase(
|
| 69 |
+
name="sql_injection_string_concat",
|
| 70 |
+
category="injection",
|
| 71 |
+
language="python",
|
| 72 |
+
code='''import sqlite3
|
| 73 |
+
|
| 74 |
+
def get_user(username):
|
| 75 |
+
conn = sqlite3.connect("users.db")
|
| 76 |
+
cursor = conn.cursor()
|
| 77 |
+
query = "SELECT * FROM users WHERE username = '" + username + "'"
|
| 78 |
+
cursor.execute(query)
|
| 79 |
+
return cursor.fetchone()
|
| 80 |
+
|
| 81 |
+
def search_products(term):
|
| 82 |
+
conn = sqlite3.connect("shop.db")
|
| 83 |
+
cursor = conn.cursor()
|
| 84 |
+
cursor.execute(f"SELECT * FROM products WHERE name LIKE '%{term}%'")
|
| 85 |
+
return cursor.fetchall()
|
| 86 |
+
''',
|
| 87 |
+
expected_findings=["injection", "sql_injection"],
|
| 88 |
+
description="SQL injection via string concatenation and f-strings",
|
| 89 |
+
),
|
| 90 |
+
BenchmarkCase(
|
| 91 |
+
name="sql_injection_format",
|
| 92 |
+
category="injection",
|
| 93 |
+
language="python",
|
| 94 |
+
code='''from django.db import connection
|
| 95 |
+
|
| 96 |
+
def lookup_user(user_id):
|
| 97 |
+
cursor = connection.cursor()
|
| 98 |
+
cursor.execute("SELECT * FROM auth_user WHERE id = %s" % user_id)
|
| 99 |
+
return cursor.fetchone()
|
| 100 |
+
|
| 101 |
+
def raw_query(table):
|
| 102 |
+
cursor = connection.cursor()
|
| 103 |
+
cursor.raw("SELECT * FROM {}".format(table))
|
| 104 |
+
''',
|
| 105 |
+
expected_findings=["injection", "sql_injection"],
|
| 106 |
+
description="SQL injection via % formatting and .format()",
|
| 107 |
+
),
|
| 108 |
+
|
| 109 |
+
# --- XSS ---
|
| 110 |
+
BenchmarkCase(
|
| 111 |
+
name="xss_dom_based",
|
| 112 |
+
category="xss",
|
| 113 |
+
language="javascript",
|
| 114 |
+
code='''function displaySearchResults() {
|
| 115 |
+
const params = new URLSearchParams(window.location.search);
|
| 116 |
+
const query = params.get('q');
|
| 117 |
+
document.getElementById('results').innerHTML = query;
|
| 118 |
+
}
|
| 119 |
+
|
| 120 |
+
function logAction(action) {
|
| 121 |
+
document.write('<p>Action: ' + action + '</p>');
|
| 122 |
+
}
|
| 123 |
+
''',
|
| 124 |
+
expected_findings=["xss"],
|
| 125 |
+
description="DOM-based XSS via innerHTML and document.write",
|
| 126 |
+
),
|
| 127 |
+
BenchmarkCase(
|
| 128 |
+
name="xss_template_injection",
|
| 129 |
+
category="xss",
|
| 130 |
+
language="python",
|
| 131 |
+
code='''from flask import Flask, request, render_template_string
|
| 132 |
+
|
| 133 |
+
app = Flask(__name__)
|
| 134 |
+
|
| 135 |
+
@app.route('/greet')
|
| 136 |
+
def greet():
|
| 137 |
+
name = request.args.get('name', 'World')
|
| 138 |
+
template = f"<h1>Hello {name}!</h1>"
|
| 139 |
+
return render_template_string(template)
|
| 140 |
+
|
| 141 |
+
@app.route('/profile')
|
| 142 |
+
def profile():
|
| 143 |
+
bio = request.args.get('bio', '')
|
| 144 |
+
return f"<div>{bio | safe}</div>"
|
| 145 |
+
''',
|
| 146 |
+
expected_findings=["xss", "injection"],
|
| 147 |
+
description="Server-side XSS via template injection and |safe filter",
|
| 148 |
+
),
|
| 149 |
+
|
| 150 |
+
# --- Command Injection ---
|
| 151 |
+
BenchmarkCase(
|
| 152 |
+
name="command_injection_os_system",
|
| 153 |
+
category="injection",
|
| 154 |
+
language="python",
|
| 155 |
+
code='''import os
|
| 156 |
+
import subprocess
|
| 157 |
+
|
| 158 |
+
def ping_host(host):
|
| 159 |
+
os.system(f"ping -c 4 {host}")
|
| 160 |
+
|
| 161 |
+
def run_tool(input_file):
|
| 162 |
+
subprocess.call(f"convert {input_file} output.png", shell=True)
|
| 163 |
+
|
| 164 |
+
def execute_command(cmd):
|
| 165 |
+
subprocess.Popen(cmd, shell=True)
|
| 166 |
+
''',
|
| 167 |
+
expected_findings=["injection", "command_injection"],
|
| 168 |
+
description="OS command injection via os.system and subprocess with shell=True",
|
| 169 |
+
),
|
| 170 |
+
|
| 171 |
+
# --- Path Traversal ---
|
| 172 |
+
BenchmarkCase(
|
| 173 |
+
name="path_traversal_open",
|
| 174 |
+
category="path-traversal",
|
| 175 |
+
language="python",
|
| 176 |
+
code='''from flask import Flask, request, send_file
|
| 177 |
+
|
| 178 |
+
app = Flask(__name__)
|
| 179 |
+
|
| 180 |
+
@app.route('/download')
|
| 181 |
+
def download_file():
|
| 182 |
+
filename = request.args.get('file')
|
| 183 |
+
with open('/var/files/' + filename, 'r') as f:
|
| 184 |
+
return f.read()
|
| 185 |
+
|
| 186 |
+
@app.route('/view')
|
| 187 |
+
def view_file():
|
| 188 |
+
path = request.args.get('path')
|
| 189 |
+
return send_file(path)
|
| 190 |
+
''',
|
| 191 |
+
expected_findings=["path-traversal"],
|
| 192 |
+
description="Path traversal via user-controlled file paths",
|
| 193 |
+
),
|
| 194 |
+
|
| 195 |
+
# --- Insecure Deserialization ---
|
| 196 |
+
BenchmarkCase(
|
| 197 |
+
name="insecure_deserialization_pickle",
|
| 198 |
+
category="deserialization",
|
| 199 |
+
language="python",
|
| 200 |
+
code='''import pickle
|
| 201 |
+
import yaml
|
| 202 |
+
|
| 203 |
+
def load_session(data):
|
| 204 |
+
return pickle.loads(data)
|
| 205 |
+
|
| 206 |
+
def load_config(content):
|
| 207 |
+
return yaml.load(content) # Missing Loader parameter
|
| 208 |
+
|
| 209 |
+
def load_cache(raw):
|
| 210 |
+
import marshal
|
| 211 |
+
return marshal.loads(raw)
|
| 212 |
+
''',
|
| 213 |
+
expected_findings=["deserialization"],
|
| 214 |
+
description="Insecure deserialization via pickle, yaml, and marshal",
|
| 215 |
+
),
|
| 216 |
+
|
| 217 |
+
# --- Weak Cryptography ---
|
| 218 |
+
BenchmarkCase(
|
| 219 |
+
name="weak_cryptography",
|
| 220 |
+
category="cryptography",
|
| 221 |
+
language="python",
|
| 222 |
+
code='''import hashlib
|
| 223 |
+
import random
|
| 224 |
+
|
| 225 |
+
def hash_password(password):
|
| 226 |
+
return hashlib.md5(password.encode()).hexdigest()
|
| 227 |
+
|
| 228 |
+
def generate_token():
|
| 229 |
+
return str(random.randint(100000, 999999))
|
| 230 |
+
|
| 231 |
+
def hash_data(data):
|
| 232 |
+
return hashlib.sha1(data.encode()).hexdigest()
|
| 233 |
+
''',
|
| 234 |
+
expected_findings=["cryptography"],
|
| 235 |
+
description="Weak cryptography: MD5, SHA1, and random module for security",
|
| 236 |
+
),
|
| 237 |
+
|
| 238 |
+
# --- Hardcoded Secrets ---
|
| 239 |
+
BenchmarkCase(
|
| 240 |
+
name="hardcoded_secrets",
|
| 241 |
+
category="credentials",
|
| 242 |
+
language="python",
|
| 243 |
+
code='''# Database configuration
|
| 244 |
+
DB_HOST = "db.prodserver.com"
|
| 245 |
+
DB_PASSWORD = "SuperSecret123!"
|
| 246 |
+
API_KEY = "PLACEHOLDER_STRIPE_KEY_FOR_TESTING_ONLY"
|
| 247 |
+
|
| 248 |
+
import requests
|
| 249 |
+
|
| 250 |
+
def call_api():
|
| 251 |
+
headers = {"Authorization": "Bearer PLACEHOLDER_JWT_TOKEN_FOR_TESTING_ONLY"}
|
| 252 |
+
return requests.get("https://api.example.com/data", headers=headers)
|
| 253 |
+
|
| 254 |
+
AWS_ACCESS_KEY = "PLACEHOLDER_AWS_KEY_FOR_TESTING_ONLY"
|
| 255 |
+
AWS_SECRET_KEY = "PLACEHOLDER_AWS_SECRET_FOR_TESTING_ONLY"
|
| 256 |
+
''',
|
| 257 |
+
expected_findings=["credentials", "cloud-credentials", "auth-tokens", "generic-secrets"],
|
| 258 |
+
description="Hardcoded passwords, API keys, JWTs, and AWS credentials",
|
| 259 |
+
),
|
| 260 |
+
|
| 261 |
+
# --- SSL/TLS Issues ---
|
| 262 |
+
BenchmarkCase(
|
| 263 |
+
name="ssl_verification_disabled",
|
| 264 |
+
category="security-misconfiguration",
|
| 265 |
+
language="python",
|
| 266 |
+
code='''import requests
|
| 267 |
+
import ssl
|
| 268 |
+
|
| 269 |
+
def fetch_data(url):
|
| 270 |
+
return requests.get(url, verify=False)
|
| 271 |
+
|
| 272 |
+
def create_context():
|
| 273 |
+
ctx = ssl._create_unverified_context()
|
| 274 |
+
return ctx
|
| 275 |
+
|
| 276 |
+
def disable_cert_check():
|
| 277 |
+
ssl._create_default_https_context = ssl._create_unverified_context
|
| 278 |
+
''',
|
| 279 |
+
expected_findings=["security-misconfiguration"],
|
| 280 |
+
description="SSL/TLS certificate verification disabled",
|
| 281 |
+
),
|
| 282 |
+
|
| 283 |
+
# --- SSRF ---
|
| 284 |
+
BenchmarkCase(
|
| 285 |
+
name="ssrf_requests",
|
| 286 |
+
category="ssrf",
|
| 287 |
+
language="python",
|
| 288 |
+
code='''import requests
|
| 289 |
+
from flask import Flask, request
|
| 290 |
+
|
| 291 |
+
app = Flask(__name__)
|
| 292 |
+
|
| 293 |
+
@app.route('/fetch')
|
| 294 |
+
def fetch_url():
|
| 295 |
+
url = request.args.get('url')
|
| 296 |
+
response = requests.get(url)
|
| 297 |
+
return response.text
|
| 298 |
+
|
| 299 |
+
@app.route('/proxy')
|
| 300 |
+
def proxy_request():
|
| 301 |
+
target = request.args.get('target')
|
| 302 |
+
return requests.post(target, data=request.form).text
|
| 303 |
+
''',
|
| 304 |
+
expected_findings=["ssrf"],
|
| 305 |
+
description="SSRF via user-controlled URL in requests",
|
| 306 |
+
),
|
| 307 |
+
|
| 308 |
+
# --- Auth Issues ---
|
| 309 |
+
BenchmarkCase(
|
| 310 |
+
name="auth_bypass",
|
| 311 |
+
category="authentication",
|
| 312 |
+
language="python",
|
| 313 |
+
code='''from flask import Flask, session
|
| 314 |
+
|
| 315 |
+
app = Flask(__name__)
|
| 316 |
+
|
| 317 |
+
@app.route('/admin')
|
| 318 |
+
def admin_panel():
|
| 319 |
+
assert session.get('is_admin')
|
| 320 |
+
return "Admin panel"
|
| 321 |
+
|
| 322 |
+
@app.route('/login', methods=['POST'])
|
| 323 |
+
def login():
|
| 324 |
+
session['authenticated'] = True
|
| 325 |
+
return "Logged in"
|
| 326 |
+
''',
|
| 327 |
+
expected_findings=["authentication"],
|
| 328 |
+
description="Auth bypass via assertion and session manipulation",
|
| 329 |
+
),
|
| 330 |
+
|
| 331 |
+
# --- Clean Code (should have minimal findings) ---
|
| 332 |
+
BenchmarkCase(
|
| 333 |
+
name="clean_code_minimal_findings",
|
| 334 |
+
category="clean",
|
| 335 |
+
language="python",
|
| 336 |
+
code='''import os
|
| 337 |
+
import hashlib
|
| 338 |
+
import secrets
|
| 339 |
+
from typing import Optional
|
| 340 |
+
|
| 341 |
+
def get_database_url() -> str:
|
| 342 |
+
"""Get database URL from environment variable."""
|
| 343 |
+
return os.environ.get("DATABASE_URL", "sqlite:///default.db")
|
| 344 |
+
|
| 345 |
+
def hash_password(password: str, salt: Optional[bytes] = None) -> str:
|
| 346 |
+
"""Hash password using SHA-256 with salt."""
|
| 347 |
+
if salt is None:
|
| 348 |
+
salt = secrets.token_bytes(32)
|
| 349 |
+
return hashlib.sha256(salt + password.encode()).hexdigest()
|
| 350 |
+
|
| 351 |
+
def generate_session_token() -> str:
|
| 352 |
+
"""Generate a cryptographically secure session token."""
|
| 353 |
+
return secrets.token_urlsafe(32)
|
| 354 |
+
|
| 355 |
+
def validate_input(data: str, max_length: int = 1000) -> str:
|
| 356 |
+
"""Validate and sanitize user input."""
|
| 357 |
+
if len(data) > max_length:
|
| 358 |
+
raise ValueError(f"Input exceeds maximum length of {max_length}")
|
| 359 |
+
return data.strip()
|
| 360 |
+
''',
|
| 361 |
+
expected_findings=[], # Clean code should have no findings
|
| 362 |
+
description="Clean, secure code that should produce minimal/no findings",
|
| 363 |
+
),
|
| 364 |
+
]
|
| 365 |
+
|
| 366 |
+
|
| 367 |
+
class BenchmarkRunner:
|
| 368 |
+
"""Run benchmark test cases against Shield Agents."""
|
| 369 |
+
|
| 370 |
+
def __init__(self, config: Optional[ShieldConfig] = None):
|
| 371 |
+
self.config = config or ShieldConfig()
|
| 372 |
+
self.results: List[BenchmarkResult] = []
|
| 373 |
+
|
| 374 |
+
async def run_all(self, category: Optional[str] = None, verbose: bool = False) -> List[BenchmarkResult]:
|
| 375 |
+
"""Run all benchmark cases.
|
| 376 |
+
|
| 377 |
+
Args:
|
| 378 |
+
category: Only run cases in this category.
|
| 379 |
+
verbose: Print detailed output.
|
| 380 |
+
|
| 381 |
+
Returns:
|
| 382 |
+
List of benchmark results.
|
| 383 |
+
"""
|
| 384 |
+
cases = BENCHMARK_CASES
|
| 385 |
+
if category:
|
| 386 |
+
cases = [c for c in cases if c.category == category]
|
| 387 |
+
|
| 388 |
+
print(f"\n{'='*60}")
|
| 389 |
+
print(f"Shield Agents Benchmark Suite")
|
| 390 |
+
print(f"Running {len(cases)} test cases...")
|
| 391 |
+
print(f"{'='*60}\n")
|
| 392 |
+
|
| 393 |
+
for case in cases:
|
| 394 |
+
result = await self.run_case(case, verbose)
|
| 395 |
+
self.results.append(result)
|
| 396 |
+
status = "PASS" if result.passed else "FAIL"
|
| 397 |
+
print(f" [{status}] {case.name} ({case.category})")
|
| 398 |
+
if not result.passed and verbose:
|
| 399 |
+
print(f" Expected: {result.expected_categories}")
|
| 400 |
+
print(f" Detected: {result.detected_categories}")
|
| 401 |
+
print(f" Missed: {result.missed_categories}")
|
| 402 |
+
if result.false_positives:
|
| 403 |
+
print(f" False +: {result.false_positives}")
|
| 404 |
+
|
| 405 |
+
return self.results
|
| 406 |
+
|
| 407 |
+
async def run_case(self, case: BenchmarkCase, verbose: bool = False) -> BenchmarkResult:
|
| 408 |
+
"""Run a single benchmark case.
|
| 409 |
+
|
| 410 |
+
Args:
|
| 411 |
+
case: Benchmark case to run.
|
| 412 |
+
verbose: Print detailed output.
|
| 413 |
+
|
| 414 |
+
Returns:
|
| 415 |
+
Benchmark result.
|
| 416 |
+
"""
|
| 417 |
+
start_time = time.time()
|
| 418 |
+
|
| 419 |
+
# Create a temporary file with the test code
|
| 420 |
+
import tempfile
|
| 421 |
+
with tempfile.NamedTemporaryFile(
|
| 422 |
+
mode="w",
|
| 423 |
+
suffix=f".{case.language == 'javascript' and 'js' or case.language}",
|
| 424 |
+
delete=False,
|
| 425 |
+
) as f:
|
| 426 |
+
f.write(case.code)
|
| 427 |
+
temp_path = f.name
|
| 428 |
+
|
| 429 |
+
try:
|
| 430 |
+
# Run scanners
|
| 431 |
+
sast = SASTScanner(self.config)
|
| 432 |
+
secrets = SecretsScanner(self.config)
|
| 433 |
+
|
| 434 |
+
sast_findings = sast.scan_file(temp_path)
|
| 435 |
+
secrets_findings = secrets.scan_file(temp_path)
|
| 436 |
+
|
| 437 |
+
all_findings = sast_findings + secrets_findings
|
| 438 |
+
|
| 439 |
+
# Extract detected categories
|
| 440 |
+
detected = set()
|
| 441 |
+
for finding in all_findings:
|
| 442 |
+
cat = finding.get("category", "").lower()
|
| 443 |
+
detected.add(cat)
|
| 444 |
+
# Also add broader categories from title
|
| 445 |
+
title = finding.get("title", "").lower()
|
| 446 |
+
if "sql" in title:
|
| 447 |
+
detected.add("sql_injection")
|
| 448 |
+
detected.add("injection")
|
| 449 |
+
if "xss" in title:
|
| 450 |
+
detected.add("xss")
|
| 451 |
+
if "command" in title or "os system" in title:
|
| 452 |
+
detected.add("command_injection")
|
| 453 |
+
detected.add("injection")
|
| 454 |
+
|
| 455 |
+
expected = set(c.lower() for c in case.expected_findings)
|
| 456 |
+
missed = expected - detected
|
| 457 |
+
false_positives = detected - expected - {"generic-secrets", "credentials"} # Allow some flexibility
|
| 458 |
+
|
| 459 |
+
# A case passes if we detect at least one expected finding
|
| 460 |
+
# (or for clean code, if we detect no expected findings)
|
| 461 |
+
if not expected:
|
| 462 |
+
# Clean code - should have no high/critical findings
|
| 463 |
+
high_severity = [f for f in all_findings if f.get("severity", "").upper() in ("CRITICAL", "HIGH")]
|
| 464 |
+
passed = len(high_severity) == 0
|
| 465 |
+
else:
|
| 466 |
+
passed = len(missed) < len(expected) # At least some expected findings detected
|
| 467 |
+
|
| 468 |
+
duration = time.time() - start_time
|
| 469 |
+
|
| 470 |
+
return BenchmarkResult(
|
| 471 |
+
name=case.name,
|
| 472 |
+
category=case.category,
|
| 473 |
+
passed=passed,
|
| 474 |
+
detected_categories=sorted(list(detected)),
|
| 475 |
+
expected_categories=sorted(list(expected)),
|
| 476 |
+
missed_categories=sorted(list(missed)),
|
| 477 |
+
false_positives=sorted(list(false_positives)),
|
| 478 |
+
findings_count=len(all_findings),
|
| 479 |
+
duration=duration,
|
| 480 |
+
)
|
| 481 |
+
|
| 482 |
+
finally:
|
| 483 |
+
os.unlink(temp_path)
|
| 484 |
+
|
| 485 |
+
def print_summary(self):
|
| 486 |
+
"""Print a summary of all benchmark results."""
|
| 487 |
+
total = len(self.results)
|
| 488 |
+
passed = sum(1 for r in self.results if r.passed)
|
| 489 |
+
failed = total - passed
|
| 490 |
+
|
| 491 |
+
print(f"\n{'='*60}")
|
| 492 |
+
print(f"Benchmark Summary")
|
| 493 |
+
print(f"{'='*60}")
|
| 494 |
+
print(f"Total: {total}")
|
| 495 |
+
print(f"Passed: {passed}")
|
| 496 |
+
print(f"Failed: {failed}")
|
| 497 |
+
print(f"Rate: {(passed/total*100):.1f}%" if total > 0 else "Rate: N/A")
|
| 498 |
+
|
| 499 |
+
if failed > 0:
|
| 500 |
+
print(f"\nFailed cases:")
|
| 501 |
+
for r in self.results:
|
| 502 |
+
if not r.passed:
|
| 503 |
+
print(f" - {r.name} (missed: {r.missed_categories})")
|
| 504 |
+
|
| 505 |
+
# Category breakdown
|
| 506 |
+
categories = {}
|
| 507 |
+
for r in self.results:
|
| 508 |
+
cat = r.category
|
| 509 |
+
if cat not in categories:
|
| 510 |
+
categories[cat] = {"passed": 0, "failed": 0}
|
| 511 |
+
if r.passed:
|
| 512 |
+
categories[cat]["passed"] += 1
|
| 513 |
+
else:
|
| 514 |
+
categories[cat]["failed"] += 1
|
| 515 |
+
|
| 516 |
+
print(f"\nCategory Breakdown:")
|
| 517 |
+
for cat, stats in sorted(categories.items()):
|
| 518 |
+
total_cat = stats["passed"] + stats["failed"]
|
| 519 |
+
print(f" {cat}: {stats['passed']}/{total_cat} passed")
|
| 520 |
+
|
| 521 |
+
return passed, total
|
| 522 |
+
|
| 523 |
+
|
| 524 |
+
async def main():
|
| 525 |
+
"""Main entry point for the benchmark suite."""
|
| 526 |
+
import argparse
|
| 527 |
+
|
| 528 |
+
parser = argparse.ArgumentParser(description="Shield Agents Benchmark Suite")
|
| 529 |
+
parser.add_argument("--verbose", "-v", action="store_true", help="Verbose output")
|
| 530 |
+
parser.add_argument("--category", "-c", help="Run only specific category")
|
| 531 |
+
parser.add_argument("--json", action="store_true", help="Output results as JSON")
|
| 532 |
+
args = parser.parse_args()
|
| 533 |
+
|
| 534 |
+
runner = BenchmarkRunner()
|
| 535 |
+
await runner.run_all(category=args.category, verbose=args.verbose)
|
| 536 |
+
runner.print_summary()
|
| 537 |
+
|
| 538 |
+
if args.json:
|
| 539 |
+
results_json = []
|
| 540 |
+
for r in runner.results:
|
| 541 |
+
results_json.append({
|
| 542 |
+
"name": r.name,
|
| 543 |
+
"category": r.category,
|
| 544 |
+
"passed": r.passed,
|
| 545 |
+
"detected": r.detected_categories,
|
| 546 |
+
"expected": r.expected_categories,
|
| 547 |
+
"missed": r.missed_categories,
|
| 548 |
+
"findings_count": r.findings_count,
|
| 549 |
+
})
|
| 550 |
+
print(json.dumps(results_json, indent=2))
|
| 551 |
+
|
| 552 |
+
|
| 553 |
+
if __name__ == "__main__":
|
| 554 |
+
asyncio.run(main())
|
config.yaml
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Shield Agents Configuration
|
| 2 |
+
# Copy this file and customize for your project
|
| 3 |
+
|
| 4 |
+
llm:
|
| 5 |
+
provider: mock # mock, openai, anthropic, ollama
|
| 6 |
+
# api_key: YOUR_API_KEY # Or set SHIELD_LLM_API_KEY env var
|
| 7 |
+
model: gpt-4
|
| 8 |
+
temperature: 0.1
|
| 9 |
+
max_tokens: 4096
|
| 10 |
+
fallback_enabled: true
|
| 11 |
+
|
| 12 |
+
scanner:
|
| 13 |
+
sast_enabled: true
|
| 14 |
+
secrets_enabled: true
|
| 15 |
+
max_file_size: 1048576 # 1MB
|
| 16 |
+
|
| 17 |
+
cache:
|
| 18 |
+
enabled: true
|
| 19 |
+
cache_dir: .shield-cache
|
| 20 |
+
incremental: true
|
| 21 |
+
max_cache_age_days: 30
|
| 22 |
+
|
| 23 |
+
deduplication:
|
| 24 |
+
enabled: true
|
| 25 |
+
similarity_threshold: 0.85
|
| 26 |
+
merge_sources: true
|
| 27 |
+
|
| 28 |
+
sarif:
|
| 29 |
+
enabled: true
|
| 30 |
+
|
| 31 |
+
shieldignore:
|
| 32 |
+
enabled: true
|
| 33 |
+
filename: .shieldignore
|
| 34 |
+
|
| 35 |
+
agents:
|
| 36 |
+
vuln_agent: true
|
| 37 |
+
threat_agent: true
|
| 38 |
+
recon_agent: true
|
| 39 |
+
compliance_agent: true
|
| 40 |
+
response_agent: true
|
| 41 |
+
autofix_agent: true
|
| 42 |
+
|
| 43 |
+
report:
|
| 44 |
+
output_dir: ./shield-reports
|
| 45 |
+
formats:
|
| 46 |
+
- html
|
| 47 |
+
- sarif
|
| 48 |
+
- json
|
| 49 |
+
include_remediation: true
|
docker-compose.yml
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version: '3.8'
|
| 2 |
+
|
| 3 |
+
services:
|
| 4 |
+
shield-agents:
|
| 5 |
+
build: .
|
| 6 |
+
container_name: shield-agents
|
| 7 |
+
volumes:
|
| 8 |
+
- ./target-code:/scan-target
|
| 9 |
+
- ./shield-reports:/app/shield-reports
|
| 10 |
+
environment:
|
| 11 |
+
- SHIELD_LLM_PROVIDER=mock
|
| 12 |
+
# - SHIELD_LLM_API_KEY=your-api-key
|
| 13 |
+
command: scan /scan-target --provider mock
|
| 14 |
+
|
| 15 |
+
# Optional: Ollama for local LLM
|
| 16 |
+
ollama:
|
| 17 |
+
image: ollama/ollama:latest
|
| 18 |
+
container_name: shield-ollama
|
| 19 |
+
ports:
|
| 20 |
+
- "11434:11434"
|
| 21 |
+
volumes:
|
| 22 |
+
- ollama_data:/root/.ollama
|
| 23 |
+
|
| 24 |
+
volumes:
|
| 25 |
+
ollama_data:
|
examples/vulnerable_app.py
ADDED
|
@@ -0,0 +1,102 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Vulnerable Example Application for Shield Agents.
|
| 3 |
+
|
| 4 |
+
This file contains intentional security vulnerabilities for testing.
|
| 5 |
+
DO NOT use this code in production!
|
| 6 |
+
"""
|
| 7 |
+
|
| 8 |
+
import os
|
| 9 |
+
import pickle
|
| 10 |
+
import hashlib
|
| 11 |
+
import random
|
| 12 |
+
import subprocess
|
| 13 |
+
import sqlite3
|
| 14 |
+
import ssl
|
| 15 |
+
import yaml
|
| 16 |
+
from flask import Flask, request, render_template_string, send_file
|
| 17 |
+
|
| 18 |
+
app = Flask(__name__)
|
| 19 |
+
|
| 20 |
+
# --- Hardcoded Secrets ---
|
| 21 |
+
DB_PASSWORD = "SuperSecret123!"
|
| 22 |
+
API_KEY = "PLACEHOLDER_STRIPE_KEY_FOR_TESTING_ONLY"
|
| 23 |
+
AWS_ACCESS_KEY = "PLACEHOLDER_AWS_KEY_FOR_TESTING_ONLY"
|
| 24 |
+
AWS_SECRET_KEY = "PLACEHOLDER_AWS_SECRET_FOR_TESTING_ONLY"
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
# --- SQL Injection ---
|
| 28 |
+
def get_user(username):
|
| 29 |
+
conn = sqlite3.connect("users.db")
|
| 30 |
+
cursor = conn.cursor()
|
| 31 |
+
query = "SELECT * FROM users WHERE username = '" + username + "'"
|
| 32 |
+
cursor.execute(query)
|
| 33 |
+
return cursor.fetchone()
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def search_products(term):
|
| 37 |
+
conn = sqlite3.connect("shop.db")
|
| 38 |
+
cursor = conn.cursor()
|
| 39 |
+
cursor.execute(f"SELECT * FROM products WHERE name LIKE '%{term}%'")
|
| 40 |
+
return cursor.fetchall()
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
# --- Command Injection ---
|
| 44 |
+
def ping_host(host):
|
| 45 |
+
os.system(f"ping -c 4 {host}")
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
def convert_image(input_file):
|
| 49 |
+
subprocess.call(f"convert {input_file} output.png", shell=True)
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
# --- Insecure Deserialization ---
|
| 53 |
+
def load_session(data):
|
| 54 |
+
return pickle.loads(data)
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def load_config(content):
|
| 58 |
+
return yaml.load(content)
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
# --- Weak Cryptography ---
|
| 62 |
+
def hash_password(password):
|
| 63 |
+
return hashlib.md5(password.encode()).hexdigest()
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
def generate_token():
|
| 67 |
+
return str(random.randint(100000, 999999))
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
# --- SSL Verification Disabled ---
|
| 71 |
+
def fetch_data(url):
|
| 72 |
+
import requests
|
| 73 |
+
return requests.get(url, verify=False)
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
# --- Path Traversal ---
|
| 77 |
+
@app.route('/download')
|
| 78 |
+
def download_file():
|
| 79 |
+
filename = request.args.get('file')
|
| 80 |
+
with open('/var/files/' + filename, 'r') as f:
|
| 81 |
+
return f.read()
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
# --- XSS / SSTI ---
|
| 85 |
+
@app.route('/greet')
|
| 86 |
+
def greet():
|
| 87 |
+
name = request.args.get('name', 'World')
|
| 88 |
+
template = f"<h1>Hello {name}!</h1>"
|
| 89 |
+
return render_template_string(template)
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
# --- SSRF ---
|
| 93 |
+
@app.route('/fetch')
|
| 94 |
+
def fetch_url():
|
| 95 |
+
url = request.args.get('url')
|
| 96 |
+
import requests
|
| 97 |
+
return requests.get(url).text
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
if __name__ == "__main__":
|
| 101 |
+
context = ssl._create_unverified_context()
|
| 102 |
+
app.run(debug=True, ssl_context=context)
|
pyproject.toml
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[build-system]
|
| 2 |
+
requires = ["setuptools>=68.0", "wheel"]
|
| 3 |
+
build-backend = "setuptools.build_meta"
|
| 4 |
+
|
| 5 |
+
[project]
|
| 6 |
+
name = "shield-agents"
|
| 7 |
+
version = "2.0.0"
|
| 8 |
+
description = "AI-Powered Multi-Agent Cybersecurity Scanner"
|
| 9 |
+
readme = "README.md"
|
| 10 |
+
license = {text = "MIT"}
|
| 11 |
+
requires-python = ">=3.8"
|
| 12 |
+
authors = [
|
| 13 |
+
{name = "Shield Agents Contributors", email = "shield-agents@proton.me"},
|
| 14 |
+
]
|
| 15 |
+
keywords = ["security", "scanner", "vulnerability", "sast", "ai", "agent", "cybersecurity", "owasp"]
|
| 16 |
+
classifiers = [
|
| 17 |
+
"Development Status :: 4 - Beta",
|
| 18 |
+
"Environment :: Console",
|
| 19 |
+
"Intended Audience :: Developers",
|
| 20 |
+
"License :: OSI Approved :: MIT License",
|
| 21 |
+
"Operating System :: OS Independent",
|
| 22 |
+
"Programming Language :: Python :: 3",
|
| 23 |
+
"Programming Language :: Python :: 3.8",
|
| 24 |
+
"Programming Language :: Python :: 3.9",
|
| 25 |
+
"Programming Language :: Python :: 3.10",
|
| 26 |
+
"Programming Language :: Python :: 3.11",
|
| 27 |
+
"Programming Language :: Python :: 3.12",
|
| 28 |
+
"Topic :: Security",
|
| 29 |
+
"Topic :: Software Development :: Quality Assurance",
|
| 30 |
+
"Topic :: Software Development :: Testing",
|
| 31 |
+
]
|
| 32 |
+
dependencies = [
|
| 33 |
+
"pyyaml>=6.0",
|
| 34 |
+
"rich>=13.0",
|
| 35 |
+
]
|
| 36 |
+
|
| 37 |
+
[project.optional-dependencies]
|
| 38 |
+
openai = ["openai>=1.0"]
|
| 39 |
+
anthropic = ["anthropic>=0.18"]
|
| 40 |
+
ollama = ["httpx>=0.24"]
|
| 41 |
+
all = ["openai>=1.0", "anthropic>=0.18", "httpx>=0.24"]
|
| 42 |
+
dev = ["pytest>=7.0", "pytest-asyncio>=0.21", "black", "ruff"]
|
| 43 |
+
|
| 44 |
+
[project.scripts]
|
| 45 |
+
shield-agents = "shield_agents.cli:main"
|
| 46 |
+
|
| 47 |
+
[project.urls]
|
| 48 |
+
Homepage = "https://github.com/shield-agents/shield-agents"
|
| 49 |
+
Documentation = "https://github.com/shield-agents/shield-agents#readme"
|
| 50 |
+
Repository = "https://github.com/shield-agents/shield-agents"
|
| 51 |
+
"Bug Tracker" = "https://github.com/shield-agents/shield-agents/issues"
|
| 52 |
+
|
| 53 |
+
[tool.pytest.ini_options]
|
| 54 |
+
testpaths = ["tests"]
|
| 55 |
+
asyncio_mode = "auto"
|
| 56 |
+
|
| 57 |
+
[tool.ruff]
|
| 58 |
+
target-version = "py38"
|
| 59 |
+
line-length = 100
|
| 60 |
+
|
| 61 |
+
[tool.ruff.lint]
|
| 62 |
+
select = ["E", "F", "W", "I"]
|
| 63 |
+
ignore = ["E501"]
|
| 64 |
+
|
| 65 |
+
[tool.black]
|
| 66 |
+
line-length = 100
|
| 67 |
+
target-version = ["py38"]
|
requirements-dev.txt
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
-r requirements.txt
|
| 2 |
+
pytest>=7.0
|
| 3 |
+
pytest-asyncio>=0.21
|
| 4 |
+
black
|
| 5 |
+
ruff
|
requirements.txt
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
pyyaml>=6.0
|
| 2 |
+
rich>=13.0
|
setup.py
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Backward-compatible setup script for Shield Agents.
|
| 2 |
+
|
| 3 |
+
Prefer using pyproject.toml for modern installations.
|
| 4 |
+
This file is kept for compatibility with older pip versions.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
from setuptools import setup
|
| 8 |
+
|
| 9 |
+
setup()
|
shield_agents/__init__.py
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Shield Agents - AI-Powered Multi-Agent Cybersecurity Scanner
|
| 3 |
+
|
| 4 |
+
A production-grade security analysis platform using coordinated AI agents
|
| 5 |
+
to detect vulnerabilities, threats, secrets, and compliance issues.
|
| 6 |
+
"""
|
| 7 |
+
|
| 8 |
+
__version__ = "2.0.0"
|
| 9 |
+
__author__ = "Shield Agents Contributors"
|
| 10 |
+
__license__ = "MIT"
|
shield_agents/agents/__init__.py
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Security analysis agents for Shield Agents."""
|
| 2 |
+
|
| 3 |
+
from .base import BaseAgent
|
| 4 |
+
from .vuln import VulnAgent
|
| 5 |
+
from .threat import ThreatAgent
|
| 6 |
+
from .recon import ReconAgent
|
| 7 |
+
from .compliance import ComplianceAgent
|
| 8 |
+
from .response import ResponseAgent
|
| 9 |
+
from .autofix import AutoFixAgent
|
| 10 |
+
|
| 11 |
+
__all__ = [
|
| 12 |
+
"BaseAgent",
|
| 13 |
+
"VulnAgent",
|
| 14 |
+
"ThreatAgent",
|
| 15 |
+
"ReconAgent",
|
| 16 |
+
"ComplianceAgent",
|
| 17 |
+
"ResponseAgent",
|
| 18 |
+
"AutoFixAgent",
|
| 19 |
+
]
|
shield_agents/agents/autofix.py
ADDED
|
@@ -0,0 +1,167 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Auto-fix / remediation agent for Shield Agents - The Master White Hat Hacker."""
|
| 2 |
+
|
| 3 |
+
import logging
|
| 4 |
+
import re
|
| 5 |
+
from typing import Any, Dict, List, Optional
|
| 6 |
+
|
| 7 |
+
from .base import BaseAgent
|
| 8 |
+
from ..config import ShieldConfig
|
| 9 |
+
from ..llm import BaseLLMProvider, create_llm_provider
|
| 10 |
+
|
| 11 |
+
logger = logging.getLogger("shield_agents.autofix_agent")
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
class AutoFixAgent(BaseAgent):
|
| 15 |
+
"""The Master White Hat Hacker agent that automatically fixes vulnerabilities.
|
| 16 |
+
|
| 17 |
+
Analyzes findings from other agents and generates:
|
| 18 |
+
1. Specific code fixes for each vulnerability
|
| 19 |
+
2. Patch files that can be applied directly
|
| 20 |
+
3. Step-by-step remediation plans
|
| 21 |
+
4. Security hardening recommendations
|
| 22 |
+
|
| 23 |
+
This agent is the crown jewel - it doesn't just find problems, it fixes them.
|
| 24 |
+
"""
|
| 25 |
+
|
| 26 |
+
# Built-in fix patterns for common vulnerabilities
|
| 27 |
+
FIX_PATTERNS = {
|
| 28 |
+
"sql_injection": {
|
| 29 |
+
"pattern": r'(cursor\.execute|\.raw)\s*\(\s*[f"\'](.*?)(?:["\']\s*\.format|\+) ',
|
| 30 |
+
"fix_template": "Use parameterized query: cursor.execute(\"SELECT ... WHERE id = %s\", (user_id,))",
|
| 31 |
+
"language": "python",
|
| 32 |
+
},
|
| 33 |
+
"eval_usage": {
|
| 34 |
+
"pattern": r'eval\s*\(',
|
| 35 |
+
"fix_template": "Replace eval() with ast.literal_eval() for safe evaluation of literals",
|
| 36 |
+
"language": "python",
|
| 37 |
+
},
|
| 38 |
+
"pickle_usage": {
|
| 39 |
+
"pattern": r'pickle\.loads?\s*\(',
|
| 40 |
+
"fix_template": "Replace pickle with JSON serialization or use pickle only with trusted data",
|
| 41 |
+
"language": "python",
|
| 42 |
+
},
|
| 43 |
+
"hardcoded_secret": {
|
| 44 |
+
"pattern": r'(password|secret|api_key|token)\s*=\s*["\'][^"\']+["\']',
|
| 45 |
+
"fix_template": "Move secrets to environment variables: os.environ.get('SECRET_KEY')",
|
| 46 |
+
"language": "python",
|
| 47 |
+
},
|
| 48 |
+
"ssl_verification": {
|
| 49 |
+
"pattern": r'verify\s*=\s*False|CERT_NONE|_create_unverified_context',
|
| 50 |
+
"fix_template": "Enable SSL verification: remove verify=False or use default SSL context",
|
| 51 |
+
"language": "python",
|
| 52 |
+
},
|
| 53 |
+
"weak_hash": {
|
| 54 |
+
"pattern": r'hashlib\.(md5|sha1)\s*\(',
|
| 55 |
+
"fix_template": "Replace with stronger hash: hashlib.sha256() or hashlib.sha512()",
|
| 56 |
+
"language": "python",
|
| 57 |
+
},
|
| 58 |
+
"unsafe_yaml": {
|
| 59 |
+
"pattern": r'yaml\.load\s*\([^)]*\)(?!.*Loader)',
|
| 60 |
+
"fix_template": "Use safe YAML loading: yaml.safe_load() or yaml.load(data, Loader=yaml.SafeLoader)",
|
| 61 |
+
"language": "python",
|
| 62 |
+
},
|
| 63 |
+
"command_injection": {
|
| 64 |
+
"pattern": r'os\.system\s*\(|subprocess\.\w+\s*\([^)]*shell\s*=\s*True',
|
| 65 |
+
"fix_template": "Use subprocess with shell=False and pass arguments as a list",
|
| 66 |
+
"language": "python",
|
| 67 |
+
},
|
| 68 |
+
"xss_dom": {
|
| 69 |
+
"pattern": r'innerHTML\s*=|document\.write\s*\(',
|
| 70 |
+
"fix_template": "Use textContent instead of innerHTML, or sanitize with DOMPurify",
|
| 71 |
+
"language": "javascript",
|
| 72 |
+
},
|
| 73 |
+
"cors_wildcard": {
|
| 74 |
+
"pattern": r'Access-Control-Allow-Origin.*\*|cors\s*=\s*True',
|
| 75 |
+
"fix_template": "Restrict CORS to specific origins: Access-Control-Allow-Origin: https://trusted-domain.com",
|
| 76 |
+
"language": "python",
|
| 77 |
+
},
|
| 78 |
+
}
|
| 79 |
+
|
| 80 |
+
def __init__(self, config: ShieldConfig, llm: Optional[BaseLLMProvider] = None):
|
| 81 |
+
super().__init__(config, llm)
|
| 82 |
+
self.name = "AutoFixAgent"
|
| 83 |
+
|
| 84 |
+
def get_system_prompt(self) -> str:
|
| 85 |
+
return (
|
| 86 |
+
"You are a master white hat hacker and security remediation expert. "
|
| 87 |
+
"Given security findings, you produce specific, copy-paste-ready code fixes. "
|
| 88 |
+
"For each finding, provide: the exact code change needed, an explanation of why "
|
| 89 |
+
"the fix works, and any additional hardening steps. "
|
| 90 |
+
"Always respond with valid JSON."
|
| 91 |
+
)
|
| 92 |
+
|
| 93 |
+
async def analyze(self, target: str, **kwargs) -> List[Dict[str, Any]]:
|
| 94 |
+
"""Generate fixes for security findings.
|
| 95 |
+
|
| 96 |
+
Args:
|
| 97 |
+
target: JSON string of findings from other agents.
|
| 98 |
+
**kwargs: Additional arguments (code_content for context).
|
| 99 |
+
|
| 100 |
+
Returns:
|
| 101 |
+
List of fix/remediation findings.
|
| 102 |
+
"""
|
| 103 |
+
code_content = kwargs.get("code_content", "")
|
| 104 |
+
|
| 105 |
+
# First try pattern-based fixes (instant, no LLM needed)
|
| 106 |
+
pattern_fixes = self._generate_pattern_fixes(target, code_content)
|
| 107 |
+
|
| 108 |
+
# Then get LLM-powered analysis for deeper fixes
|
| 109 |
+
llm_fixes = []
|
| 110 |
+
if self.config.llm.provider != "mock" or pattern_fixes == []:
|
| 111 |
+
result = await self.llm.complete_json([
|
| 112 |
+
{"role": "system", "content": self.get_system_prompt()},
|
| 113 |
+
{"role": "user", "content": self._build_fix_prompt(target, code_content)},
|
| 114 |
+
])
|
| 115 |
+
llm_fixes = result.get("findings", result.get("fixes", []))
|
| 116 |
+
|
| 117 |
+
all_fixes = pattern_fixes + llm_fixes
|
| 118 |
+
for fix in all_fixes:
|
| 119 |
+
fix["agent"] = self.name
|
| 120 |
+
fix["source"] = self.name
|
| 121 |
+
fix.setdefault("id", f"{self.name}-{len(self.findings) + 1}")
|
| 122 |
+
fix.setdefault("severity", "INFO")
|
| 123 |
+
fix.setdefault("category", "remediation")
|
| 124 |
+
fix.setdefault("confidence", 0.9)
|
| 125 |
+
self.add_finding(fix)
|
| 126 |
+
|
| 127 |
+
logger.info(f"{self.name}: Generated {len(all_fixes)} fixes/remediations")
|
| 128 |
+
return all_fixes
|
| 129 |
+
|
| 130 |
+
def _generate_pattern_fixes(self, findings_json: str, code_content: str) -> List[Dict[str, Any]]:
|
| 131 |
+
"""Generate fixes using built-in patterns - instant, no LLM needed."""
|
| 132 |
+
fixes = []
|
| 133 |
+
|
| 134 |
+
for vuln_type, fix_info in self.FIX_PATTERNS.items():
|
| 135 |
+
try:
|
| 136 |
+
matches = list(re.finditer(fix_info["pattern"], code_content, re.IGNORECASE))
|
| 137 |
+
for match in matches:
|
| 138 |
+
line_num = code_content[:match.start()].count("\n") + 1
|
| 139 |
+
fixes.append({
|
| 140 |
+
"title": f"Auto-fix: {vuln_type.replace('_', ' ').title()}",
|
| 141 |
+
"description": f"Automatically detected and generated fix for {vuln_type.replace('_', ' ')}",
|
| 142 |
+
"vulnerability_type": vuln_type,
|
| 143 |
+
"line": line_num,
|
| 144 |
+
"code_snippet": match.group(0),
|
| 145 |
+
"fix": fix_info["fix_template"],
|
| 146 |
+
"language": fix_info["language"],
|
| 147 |
+
"fix_type": "pattern_based",
|
| 148 |
+
"auto_applicable": True,
|
| 149 |
+
"severity": "INFO",
|
| 150 |
+
})
|
| 151 |
+
except re.error:
|
| 152 |
+
continue
|
| 153 |
+
|
| 154 |
+
return fixes
|
| 155 |
+
|
| 156 |
+
def _build_fix_prompt(self, findings_json: str, code_content: str = "") -> str:
|
| 157 |
+
prompt = (
|
| 158 |
+
f"Given these security findings:\n\n{findings_json}\n\n"
|
| 159 |
+
)
|
| 160 |
+
if code_content:
|
| 161 |
+
prompt += f"And the following source code:\n\n```\n{code_content}\n```\n\n"
|
| 162 |
+
prompt += (
|
| 163 |
+
"Generate specific, copy-paste-ready code fixes for each finding. "
|
| 164 |
+
"Return JSON with 'fixes' array. Each fix: title, description, "
|
| 165 |
+
"vulnerability_type, original_code, fixed_code, explanation, severity."
|
| 166 |
+
)
|
| 167 |
+
return prompt
|
shield_agents/agents/base.py
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Base agent class for all Shield Agents."""
|
| 2 |
+
|
| 3 |
+
import logging
|
| 4 |
+
from abc import ABC, abstractmethod
|
| 5 |
+
from typing import Any, Dict, List, Optional
|
| 6 |
+
|
| 7 |
+
from ..config import ShieldConfig
|
| 8 |
+
from ..llm import BaseLLMProvider, create_llm_provider
|
| 9 |
+
|
| 10 |
+
logger = logging.getLogger("shield_agents.agents")
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
class BaseAgent(ABC):
|
| 14 |
+
"""Abstract base class for all security analysis agents.
|
| 15 |
+
|
| 16 |
+
Each agent specializes in a particular aspect of security analysis
|
| 17 |
+
and communicates findings through a standardized interface.
|
| 18 |
+
"""
|
| 19 |
+
|
| 20 |
+
def __init__(self, config: ShieldConfig, llm: Optional[BaseLLMProvider] = None):
|
| 21 |
+
self.config = config
|
| 22 |
+
self.llm = llm or create_llm_provider(config.llm)
|
| 23 |
+
self.name = self.__class__.__name__
|
| 24 |
+
self.findings: List[Dict[str, Any]] = []
|
| 25 |
+
|
| 26 |
+
@abstractmethod
|
| 27 |
+
async def analyze(self, target: str, **kwargs) -> List[Dict[str, Any]]:
|
| 28 |
+
"""Analyze a target and return findings.
|
| 29 |
+
|
| 30 |
+
Args:
|
| 31 |
+
target: Path or content to analyze.
|
| 32 |
+
**kwargs: Additional arguments.
|
| 33 |
+
|
| 34 |
+
Returns:
|
| 35 |
+
List of finding dictionaries.
|
| 36 |
+
"""
|
| 37 |
+
pass
|
| 38 |
+
|
| 39 |
+
@abstractmethod
|
| 40 |
+
def get_system_prompt(self) -> str:
|
| 41 |
+
"""Return the system prompt for this agent's LLM interactions."""
|
| 42 |
+
pass
|
| 43 |
+
|
| 44 |
+
def add_finding(self, finding: Dict[str, Any]) -> None:
|
| 45 |
+
"""Add a finding to the agent's results.
|
| 46 |
+
|
| 47 |
+
Args:
|
| 48 |
+
finding: Finding dictionary with standardized fields.
|
| 49 |
+
"""
|
| 50 |
+
# Ensure required fields
|
| 51 |
+
finding.setdefault("id", f"{self.name}-{len(self.findings) + 1}")
|
| 52 |
+
finding.setdefault("agent", self.name)
|
| 53 |
+
finding.setdefault("severity", "MEDIUM")
|
| 54 |
+
finding.setdefault("category", "unknown")
|
| 55 |
+
finding.setdefault("source", self.name)
|
| 56 |
+
finding.setdefault("confidence", 0.8)
|
| 57 |
+
self.findings.append(finding)
|
| 58 |
+
|
| 59 |
+
def clear_findings(self) -> None:
|
| 60 |
+
"""Clear all findings."""
|
| 61 |
+
self.findings = []
|
| 62 |
+
|
| 63 |
+
def get_findings(self) -> List[Dict[str, Any]]:
|
| 64 |
+
"""Get all findings."""
|
| 65 |
+
return self.findings
|
| 66 |
+
|
| 67 |
+
def _build_analysis_prompt(self, code_content: str, file_path: str = "") -> str:
|
| 68 |
+
"""Build a user prompt for LLM analysis.
|
| 69 |
+
|
| 70 |
+
Args:
|
| 71 |
+
code_content: Code to analyze.
|
| 72 |
+
file_path: Path to the file being analyzed.
|
| 73 |
+
|
| 74 |
+
Returns:
|
| 75 |
+
Formatted prompt string.
|
| 76 |
+
"""
|
| 77 |
+
return (
|
| 78 |
+
f"Analyze the following code for security vulnerabilities.\n\n"
|
| 79 |
+
f"File: {file_path or 'unknown'}\n\n"
|
| 80 |
+
f"```\n{code_content}\n```\n\n"
|
| 81 |
+
f"Return a JSON object with a 'findings' array. Each finding should have:\n"
|
| 82 |
+
f"- title: Short description of the issue\n"
|
| 83 |
+
f"- description: Detailed explanation\n"
|
| 84 |
+
f"- severity: CRITICAL, HIGH, MEDIUM, LOW, or INFO\n"
|
| 85 |
+
f"- category: Vulnerability category (e.g., injection, xss, auth)\n"
|
| 86 |
+
f"- line: Line number where the issue was found\n"
|
| 87 |
+
f"- code_snippet: The problematic code\n"
|
| 88 |
+
f"- remediation: How to fix the issue\n"
|
| 89 |
+
f"- cwe: CWE identifier if applicable\n"
|
| 90 |
+
)
|
shield_agents/agents/compliance.py
ADDED
|
@@ -0,0 +1,74 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Compliance checking agent for Shield Agents."""
|
| 2 |
+
|
| 3 |
+
import logging
|
| 4 |
+
from typing import Any, Dict, List, Optional
|
| 5 |
+
|
| 6 |
+
from .base import BaseAgent
|
| 7 |
+
from ..config import ShieldConfig
|
| 8 |
+
from ..llm import BaseLLMProvider, create_llm_provider
|
| 9 |
+
from ..knowledge.owasp import get_owasp_category
|
| 10 |
+
|
| 11 |
+
logger = logging.getLogger("shield_agents.compliance_agent")
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
class ComplianceAgent(BaseAgent):
|
| 15 |
+
"""Agent specialized in compliance and security standards checking.
|
| 16 |
+
|
| 17 |
+
Checks code against OWASP Top 10, SANS Top 25, and other security standards.
|
| 18 |
+
"""
|
| 19 |
+
|
| 20 |
+
def __init__(self, config: ShieldConfig, llm: Optional[BaseLLMProvider] = None):
|
| 21 |
+
super().__init__(config, llm)
|
| 22 |
+
self.name = "ComplianceAgent"
|
| 23 |
+
|
| 24 |
+
def get_system_prompt(self) -> str:
|
| 25 |
+
return (
|
| 26 |
+
"You are a security compliance expert. Check code against OWASP Top 10 2021, "
|
| 27 |
+
"SANS Top 25, and industry security standards. Identify compliance violations "
|
| 28 |
+
"and provide specific standard references. "
|
| 29 |
+
"Always respond with valid JSON containing a 'findings' array."
|
| 30 |
+
)
|
| 31 |
+
|
| 32 |
+
async def analyze(self, target: str, **kwargs) -> List[Dict[str, Any]]:
|
| 33 |
+
"""Analyze code for compliance violations.
|
| 34 |
+
|
| 35 |
+
Args:
|
| 36 |
+
target: Code content to analyze.
|
| 37 |
+
**kwargs: Additional arguments (file_path).
|
| 38 |
+
|
| 39 |
+
Returns:
|
| 40 |
+
List of compliance findings.
|
| 41 |
+
"""
|
| 42 |
+
file_path = kwargs.get("file_path", "unknown")
|
| 43 |
+
|
| 44 |
+
result = await self.llm.complete_json([
|
| 45 |
+
{"role": "system", "content": self.get_system_prompt()},
|
| 46 |
+
{"role": "user", "content": self._build_compliance_prompt(target, file_path)},
|
| 47 |
+
])
|
| 48 |
+
|
| 49 |
+
findings = result.get("findings", [])
|
| 50 |
+
for finding in findings:
|
| 51 |
+
finding["agent"] = self.name
|
| 52 |
+
finding["source"] = self.name
|
| 53 |
+
finding["file"] = file_path
|
| 54 |
+
finding.setdefault("id", f"{self.name}-{len(self.findings) + 1}")
|
| 55 |
+
finding.setdefault("severity", "MEDIUM")
|
| 56 |
+
finding.setdefault("category", "compliance")
|
| 57 |
+
finding.setdefault("confidence", 0.75)
|
| 58 |
+
# Add OWASP mapping
|
| 59 |
+
category = finding.get("category", "")
|
| 60 |
+
owasp_info = get_owasp_category(category)
|
| 61 |
+
finding["owasp"] = owasp_info.get("name", "Unknown")
|
| 62 |
+
self.add_finding(finding)
|
| 63 |
+
|
| 64 |
+
logger.info(f"{self.name}: Found {len(findings)} compliance issues in {file_path}")
|
| 65 |
+
return findings
|
| 66 |
+
|
| 67 |
+
def _build_compliance_prompt(self, code_content: str, file_path: str = "") -> str:
|
| 68 |
+
return (
|
| 69 |
+
f"Check the following code for compliance with OWASP Top 10 2021 and security standards.\n\n"
|
| 70 |
+
f"File: {file_path}\n\n"
|
| 71 |
+
f"```\n{code_content}\n```\n\n"
|
| 72 |
+
f"Return JSON with 'findings' array. Each finding: title, description, severity, "
|
| 73 |
+
f"category (use owasp category names), owasp_reference, line, remediation."
|
| 74 |
+
)
|
shield_agents/agents/recon.py
ADDED
|
@@ -0,0 +1,69 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Reconnaissance agent for Shield Agents."""
|
| 2 |
+
|
| 3 |
+
import logging
|
| 4 |
+
from typing import Any, Dict, List, Optional
|
| 5 |
+
|
| 6 |
+
from .base import BaseAgent
|
| 7 |
+
from ..config import ShieldConfig
|
| 8 |
+
from ..llm import BaseLLMProvider, create_llm_provider
|
| 9 |
+
|
| 10 |
+
logger = logging.getLogger("shield_agents.recon_agent")
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
class ReconAgent(BaseAgent):
|
| 14 |
+
"""Agent specialized in reconnaissance and information gathering.
|
| 15 |
+
|
| 16 |
+
Identifies exposed information, debug endpoints, API keys, comments
|
| 17 |
+
containing sensitive data, and other information disclosure issues.
|
| 18 |
+
"""
|
| 19 |
+
|
| 20 |
+
def __init__(self, config: ShieldConfig, llm: Optional[BaseLLMProvider] = None):
|
| 21 |
+
super().__init__(config, llm)
|
| 22 |
+
self.name = "ReconAgent"
|
| 23 |
+
|
| 24 |
+
def get_system_prompt(self) -> str:
|
| 25 |
+
return (
|
| 26 |
+
"You are a security reconnaissance expert. Analyze code for information disclosure, "
|
| 27 |
+
"exposed debug endpoints, sensitive comments, metadata leaks, and other recon findings. "
|
| 28 |
+
"Always respond with valid JSON containing a 'findings' array."
|
| 29 |
+
)
|
| 30 |
+
|
| 31 |
+
async def analyze(self, target: str, **kwargs) -> List[Dict[str, Any]]:
|
| 32 |
+
"""Analyze code for information disclosure.
|
| 33 |
+
|
| 34 |
+
Args:
|
| 35 |
+
target: Code content to analyze.
|
| 36 |
+
**kwargs: Additional arguments (file_path).
|
| 37 |
+
|
| 38 |
+
Returns:
|
| 39 |
+
List of reconnaissance findings.
|
| 40 |
+
"""
|
| 41 |
+
file_path = kwargs.get("file_path", "unknown")
|
| 42 |
+
|
| 43 |
+
result = await self.llm.complete_json([
|
| 44 |
+
{"role": "system", "content": self.get_system_prompt()},
|
| 45 |
+
{"role": "user", "content": self._build_recon_prompt(target, file_path)},
|
| 46 |
+
])
|
| 47 |
+
|
| 48 |
+
findings = result.get("findings", [])
|
| 49 |
+
for finding in findings:
|
| 50 |
+
finding["agent"] = self.name
|
| 51 |
+
finding["source"] = self.name
|
| 52 |
+
finding["file"] = file_path
|
| 53 |
+
finding.setdefault("id", f"{self.name}-{len(self.findings) + 1}")
|
| 54 |
+
finding.setdefault("severity", "LOW")
|
| 55 |
+
finding.setdefault("category", "information-disclosure")
|
| 56 |
+
finding.setdefault("confidence", 0.75)
|
| 57 |
+
self.add_finding(finding)
|
| 58 |
+
|
| 59 |
+
logger.info(f"{self.name}: Found {len(findings)} recon findings in {file_path}")
|
| 60 |
+
return findings
|
| 61 |
+
|
| 62 |
+
def _build_recon_prompt(self, code_content: str, file_path: str = "") -> str:
|
| 63 |
+
return (
|
| 64 |
+
f"Analyze the following code for information disclosure and recon findings.\n\n"
|
| 65 |
+
f"File: {file_path}\n\n"
|
| 66 |
+
f"```\n{code_content}\n```\n\n"
|
| 67 |
+
f"Return JSON with 'findings' array. Each finding: title, description, severity, "
|
| 68 |
+
f"category, line, code_snippet, remediation."
|
| 69 |
+
)
|
shield_agents/agents/response.py
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Incident response agent for Shield Agents."""
|
| 2 |
+
|
| 3 |
+
import logging
|
| 4 |
+
from typing import Any, Dict, List, Optional
|
| 5 |
+
|
| 6 |
+
from .base import BaseAgent
|
| 7 |
+
from ..config import ShieldConfig
|
| 8 |
+
from ..llm import BaseLLMProvider, create_llm_provider
|
| 9 |
+
|
| 10 |
+
logger = logging.getLogger("shield_agents.response_agent")
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
class ResponseAgent(BaseAgent):
|
| 14 |
+
"""Agent specialized in incident response and risk assessment.
|
| 15 |
+
|
| 16 |
+
Provides risk scoring, incident response recommendations, and
|
| 17 |
+
prioritized remediation guidance based on findings from other agents.
|
| 18 |
+
"""
|
| 19 |
+
|
| 20 |
+
def __init__(self, config: ShieldConfig, llm: Optional[BaseLLMProvider] = None):
|
| 21 |
+
super().__init__(config, llm)
|
| 22 |
+
self.name = "ResponseAgent"
|
| 23 |
+
|
| 24 |
+
def get_system_prompt(self) -> str:
|
| 25 |
+
return (
|
| 26 |
+
"You are an incident response expert. Analyze security findings and provide "
|
| 27 |
+
"risk assessment, prioritized remediation plans, and incident response procedures. "
|
| 28 |
+
"Focus on practical, actionable recommendations. "
|
| 29 |
+
"Always respond with valid JSON."
|
| 30 |
+
)
|
| 31 |
+
|
| 32 |
+
async def analyze(self, target: str, **kwargs) -> List[Dict[str, Any]]:
|
| 33 |
+
"""Analyze findings and generate response recommendations.
|
| 34 |
+
|
| 35 |
+
Args:
|
| 36 |
+
target: JSON string of findings from other agents.
|
| 37 |
+
**kwargs: Additional arguments.
|
| 38 |
+
|
| 39 |
+
Returns:
|
| 40 |
+
List of response/recommendation findings.
|
| 41 |
+
"""
|
| 42 |
+
result = await self.llm.complete_json([
|
| 43 |
+
{"role": "system", "content": self.get_system_prompt()},
|
| 44 |
+
{"role": "user", "content": self._build_response_prompt(target)},
|
| 45 |
+
])
|
| 46 |
+
|
| 47 |
+
findings = result.get("findings", result.get("recommendations", []))
|
| 48 |
+
for finding in findings:
|
| 49 |
+
finding["agent"] = self.name
|
| 50 |
+
finding["source"] = self.name
|
| 51 |
+
finding.setdefault("id", f"{self.name}-{len(self.findings) + 1}")
|
| 52 |
+
finding.setdefault("severity", "MEDIUM")
|
| 53 |
+
finding.setdefault("category", "incident-response")
|
| 54 |
+
finding.setdefault("confidence", 0.85)
|
| 55 |
+
self.add_finding(finding)
|
| 56 |
+
|
| 57 |
+
logger.info(f"{self.name}: Generated {len(findings)} response recommendations")
|
| 58 |
+
return findings
|
| 59 |
+
|
| 60 |
+
def _build_response_prompt(self, findings_json: str) -> str:
|
| 61 |
+
return (
|
| 62 |
+
f"Based on the following security findings, provide risk assessment and "
|
| 63 |
+
f"prioritized remediation recommendations:\n\n"
|
| 64 |
+
f"{findings_json}\n\n"
|
| 65 |
+
f"Return JSON with 'recommendations' array. Each: title, description, severity, "
|
| 66 |
+
f"priority (1-5), action_steps (list), affected_findings (list of IDs)."
|
| 67 |
+
)
|
shield_agents/agents/threat.py
ADDED
|
@@ -0,0 +1,69 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Threat modeling agent for Shield Agents."""
|
| 2 |
+
|
| 3 |
+
import logging
|
| 4 |
+
from typing import Any, Dict, List, Optional
|
| 5 |
+
|
| 6 |
+
from .base import BaseAgent
|
| 7 |
+
from ..config import ShieldConfig
|
| 8 |
+
from ..llm import BaseLLMProvider, create_llm_provider
|
| 9 |
+
|
| 10 |
+
logger = logging.getLogger("shield_agents.threat_agent")
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
class ThreatAgent(BaseAgent):
|
| 14 |
+
"""Agent specialized in threat modeling and attack surface analysis.
|
| 15 |
+
|
| 16 |
+
Identifies potential attack vectors, threat actors, and attack scenarios.
|
| 17 |
+
"""
|
| 18 |
+
|
| 19 |
+
def __init__(self, config: ShieldConfig, llm: Optional[BaseLLMProvider] = None):
|
| 20 |
+
super().__init__(config, llm)
|
| 21 |
+
self.name = "ThreatAgent"
|
| 22 |
+
|
| 23 |
+
def get_system_prompt(self) -> str:
|
| 24 |
+
return (
|
| 25 |
+
"You are a threat modeling expert. Analyze code from an attacker's perspective. "
|
| 26 |
+
"Identify attack vectors, threat scenarios, and potential exploits. "
|
| 27 |
+
"Map findings to MITRE ATT&CK techniques where applicable. "
|
| 28 |
+
"Always respond with valid JSON containing a 'findings' array."
|
| 29 |
+
)
|
| 30 |
+
|
| 31 |
+
async def analyze(self, target: str, **kwargs) -> List[Dict[str, Any]]:
|
| 32 |
+
"""Analyze code for threat vectors.
|
| 33 |
+
|
| 34 |
+
Args:
|
| 35 |
+
target: Code content to analyze.
|
| 36 |
+
**kwargs: Additional arguments (file_path).
|
| 37 |
+
|
| 38 |
+
Returns:
|
| 39 |
+
List of threat findings.
|
| 40 |
+
"""
|
| 41 |
+
file_path = kwargs.get("file_path", "unknown")
|
| 42 |
+
|
| 43 |
+
result = await self.llm.complete_json([
|
| 44 |
+
{"role": "system", "content": self.get_system_prompt()},
|
| 45 |
+
{"role": "user", "content": self._build_threat_prompt(target, file_path)},
|
| 46 |
+
])
|
| 47 |
+
|
| 48 |
+
findings = result.get("findings", [])
|
| 49 |
+
for finding in findings:
|
| 50 |
+
finding["agent"] = self.name
|
| 51 |
+
finding["source"] = self.name
|
| 52 |
+
finding["file"] = file_path
|
| 53 |
+
finding.setdefault("id", f"{self.name}-{len(self.findings) + 1}")
|
| 54 |
+
finding.setdefault("severity", "MEDIUM")
|
| 55 |
+
finding.setdefault("category", "threat")
|
| 56 |
+
finding.setdefault("confidence", 0.7)
|
| 57 |
+
self.add_finding(finding)
|
| 58 |
+
|
| 59 |
+
logger.info(f"{self.name}: Found {len(findings)} threat vectors in {file_path}")
|
| 60 |
+
return findings
|
| 61 |
+
|
| 62 |
+
def _build_threat_prompt(self, code_content: str, file_path: str = "") -> str:
|
| 63 |
+
return (
|
| 64 |
+
f"Analyze the following code for potential attack vectors and threats.\n\n"
|
| 65 |
+
f"File: {file_path}\n\n"
|
| 66 |
+
f"```\n{code_content}\n```\n\n"
|
| 67 |
+
f"Return JSON with 'findings' array. Each finding: title, description, severity, "
|
| 68 |
+
f"attack_vector, mitre_technique, remediation."
|
| 69 |
+
)
|
shield_agents/agents/vuln.py
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Vulnerability detection agent for Shield Agents."""
|
| 2 |
+
|
| 3 |
+
import logging
|
| 4 |
+
from typing import Any, Dict, List, Optional
|
| 5 |
+
|
| 6 |
+
from .base import BaseAgent
|
| 7 |
+
from ..config import ShieldConfig
|
| 8 |
+
from ..llm import BaseLLMProvider, create_llm_provider
|
| 9 |
+
|
| 10 |
+
logger = logging.getLogger("shield_agents.vuln_agent")
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
class VulnAgent(BaseAgent):
|
| 14 |
+
"""Agent specialized in detecting code vulnerabilities.
|
| 15 |
+
|
| 16 |
+
Uses LLM analysis and pattern matching to identify security
|
| 17 |
+
vulnerabilities like SQL injection, XSS, command injection, etc.
|
| 18 |
+
"""
|
| 19 |
+
|
| 20 |
+
def __init__(self, config: ShieldConfig, llm: Optional[BaseLLMProvider] = None):
|
| 21 |
+
super().__init__(config, llm)
|
| 22 |
+
self.name = "VulnAgent"
|
| 23 |
+
|
| 24 |
+
def get_system_prompt(self) -> str:
|
| 25 |
+
return (
|
| 26 |
+
"You are an expert security analyst specializing in vulnerability detection. "
|
| 27 |
+
"Your task is to identify security vulnerabilities in code. "
|
| 28 |
+
"Focus on: SQL injection, XSS, command injection, path traversal, "
|
| 29 |
+
"insecure deserialization, authentication bypass, and other common vulnerabilities. "
|
| 30 |
+
"For each finding, provide the severity, category, line number, and remediation advice. "
|
| 31 |
+
"Always respond with valid JSON containing a 'findings' array."
|
| 32 |
+
)
|
| 33 |
+
|
| 34 |
+
async def analyze(self, target: str, **kwargs) -> List[Dict[str, Any]]:
|
| 35 |
+
"""Analyze code content for vulnerabilities.
|
| 36 |
+
|
| 37 |
+
Args:
|
| 38 |
+
target: Code content to analyze.
|
| 39 |
+
**kwargs: Additional arguments (file_path).
|
| 40 |
+
|
| 41 |
+
Returns:
|
| 42 |
+
List of vulnerability findings.
|
| 43 |
+
"""
|
| 44 |
+
file_path = kwargs.get("file_path", "unknown")
|
| 45 |
+
|
| 46 |
+
if self.config.llm.provider == "mock":
|
| 47 |
+
# Use smart mock provider which does pattern matching
|
| 48 |
+
result = await self.llm.complete_json([
|
| 49 |
+
{"role": "system", "content": self.get_system_prompt()},
|
| 50 |
+
{"role": "user", "content": self._build_analysis_prompt(target, file_path)},
|
| 51 |
+
])
|
| 52 |
+
else:
|
| 53 |
+
# Use real LLM with fallback handling
|
| 54 |
+
result = await self.llm.complete_json([
|
| 55 |
+
{"role": "system", "content": self.get_system_prompt()},
|
| 56 |
+
{"role": "user", "content": self._build_analysis_prompt(target, file_path)},
|
| 57 |
+
])
|
| 58 |
+
|
| 59 |
+
findings = result.get("findings", [])
|
| 60 |
+
for finding in findings:
|
| 61 |
+
finding["agent"] = self.name
|
| 62 |
+
finding["source"] = self.name
|
| 63 |
+
finding["file"] = file_path
|
| 64 |
+
finding.setdefault("id", f"{self.name}-{len(self.findings) + 1}")
|
| 65 |
+
finding.setdefault("severity", "MEDIUM")
|
| 66 |
+
finding.setdefault("category", "vulnerability")
|
| 67 |
+
finding.setdefault("confidence", 0.8)
|
| 68 |
+
self.add_finding(finding)
|
| 69 |
+
|
| 70 |
+
logger.info(f"{self.name}: Found {len(findings)} vulnerabilities in {file_path}")
|
| 71 |
+
return findings
|
| 72 |
+
|
| 73 |
+
def _build_analysis_prompt(self, code_content: str, file_path: str = "") -> str:
|
| 74 |
+
return (
|
| 75 |
+
f"Analyze the following code for security vulnerabilities.\n\n"
|
| 76 |
+
f"File: {file_path}\n\n"
|
| 77 |
+
f"```\n{code_content}\n```\n\n"
|
| 78 |
+
f"Return JSON with 'findings' array. Each finding: title, description, severity, "
|
| 79 |
+
f"category, line, code_snippet, remediation, cwe."
|
| 80 |
+
)
|
shield_agents/cache.py
ADDED
|
@@ -0,0 +1,238 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Caching and Incremental Scanning for Shield Agents.
|
| 3 |
+
|
| 4 |
+
Cache previous scan results and only re-scan changed files.
|
| 5 |
+
This is the difference between a demo and a production tool.
|
| 6 |
+
"""
|
| 7 |
+
|
| 8 |
+
import json
|
| 9 |
+
import logging
|
| 10 |
+
import os
|
| 11 |
+
import time
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
from typing import Any, Dict, List, Optional, Set
|
| 14 |
+
|
| 15 |
+
from .utils.crypto import hash_file
|
| 16 |
+
|
| 17 |
+
logger = logging.getLogger("shield_agents.cache")
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
class ScanCache:
|
| 21 |
+
"""Manages scan result caching for incremental scanning.
|
| 22 |
+
|
| 23 |
+
The cache stores:
|
| 24 |
+
- File hashes (to detect changes)
|
| 25 |
+
- Scan results per file
|
| 26 |
+
- Timestamps (for cache invalidation)
|
| 27 |
+
- Global metadata
|
| 28 |
+
"""
|
| 29 |
+
|
| 30 |
+
def __init__(self, cache_dir: str = ".shield-cache", max_age_days: int = 30):
|
| 31 |
+
"""Initialize the scan cache.
|
| 32 |
+
|
| 33 |
+
Args:
|
| 34 |
+
cache_dir: Directory for cache storage.
|
| 35 |
+
max_age_days: Maximum age of cache entries in days.
|
| 36 |
+
"""
|
| 37 |
+
self.cache_dir = Path(cache_dir)
|
| 38 |
+
self.max_age_days = max_age_days
|
| 39 |
+
self.cache_file = self.cache_dir / "scan_cache.json"
|
| 40 |
+
self._cache: Dict[str, Any] = {}
|
| 41 |
+
self._loaded = False
|
| 42 |
+
self.changed = False
|
| 43 |
+
|
| 44 |
+
def _ensure_loaded(self) -> None:
|
| 45 |
+
"""Load cache from disk if not already loaded."""
|
| 46 |
+
if not self._loaded:
|
| 47 |
+
self._load()
|
| 48 |
+
self._loaded = True
|
| 49 |
+
|
| 50 |
+
def _load(self) -> None:
|
| 51 |
+
"""Load cache from disk."""
|
| 52 |
+
if self.cache_file.exists():
|
| 53 |
+
try:
|
| 54 |
+
with open(self.cache_file, "r") as f:
|
| 55 |
+
self._cache = json.load(f)
|
| 56 |
+
logger.debug(f"Loaded cache with {len(self._cache.get('files', {}))} entries")
|
| 57 |
+
except (json.JSONDecodeError, IOError) as e:
|
| 58 |
+
logger.warning(f"Failed to load cache: {e}")
|
| 59 |
+
self._cache = {"files": {}, "metadata": {}}
|
| 60 |
+
else:
|
| 61 |
+
self._cache = {"files": {}, "metadata": {}}
|
| 62 |
+
|
| 63 |
+
def save(self) -> None:
|
| 64 |
+
"""Save cache to disk."""
|
| 65 |
+
if not self.changed:
|
| 66 |
+
return
|
| 67 |
+
|
| 68 |
+
self.cache_dir.mkdir(parents=True, exist_ok=True)
|
| 69 |
+
self._cache["metadata"]["last_saved"] = time.time()
|
| 70 |
+
|
| 71 |
+
try:
|
| 72 |
+
with open(self.cache_file, "w") as f:
|
| 73 |
+
json.dump(self._cache, f, indent=2)
|
| 74 |
+
self.changed = False
|
| 75 |
+
logger.debug("Cache saved successfully")
|
| 76 |
+
except IOError as e:
|
| 77 |
+
logger.warning(f"Failed to save cache: {e}")
|
| 78 |
+
|
| 79 |
+
def get_file_hash(self, file_path: str) -> Optional[str]:
|
| 80 |
+
"""Get the cached hash for a file.
|
| 81 |
+
|
| 82 |
+
Args:
|
| 83 |
+
file_path: Path to the file.
|
| 84 |
+
|
| 85 |
+
Returns:
|
| 86 |
+
Cached hash, or None if not cached.
|
| 87 |
+
"""
|
| 88 |
+
self._ensure_loaded()
|
| 89 |
+
return self._cache.get("files", {}).get(file_path, {}).get("hash")
|
| 90 |
+
|
| 91 |
+
def compute_file_hash(self, file_path: str) -> Optional[str]:
|
| 92 |
+
"""Compute current hash for a file.
|
| 93 |
+
|
| 94 |
+
Args:
|
| 95 |
+
file_path: Path to the file.
|
| 96 |
+
|
| 97 |
+
Returns:
|
| 98 |
+
Current file hash, or None if file cannot be read.
|
| 99 |
+
"""
|
| 100 |
+
return hash_file(file_path)
|
| 101 |
+
|
| 102 |
+
def is_file_changed(self, file_path: str) -> bool:
|
| 103 |
+
"""Check if a file has changed since last scan.
|
| 104 |
+
|
| 105 |
+
Args:
|
| 106 |
+
file_path: Path to the file.
|
| 107 |
+
|
| 108 |
+
Returns:
|
| 109 |
+
True if file is new or modified, False if unchanged.
|
| 110 |
+
"""
|
| 111 |
+
self._ensure_loaded()
|
| 112 |
+
current_hash = self.compute_file_hash(file_path)
|
| 113 |
+
cached_hash = self.get_file_hash(file_path)
|
| 114 |
+
|
| 115 |
+
if current_hash is None:
|
| 116 |
+
# Can't read the file, treat as unchanged
|
| 117 |
+
return False
|
| 118 |
+
|
| 119 |
+
if cached_hash is None:
|
| 120 |
+
# Not cached, treat as new
|
| 121 |
+
return True
|
| 122 |
+
|
| 123 |
+
return current_hash != cached_hash
|
| 124 |
+
|
| 125 |
+
def get_changed_files(self, file_paths: List[str]) -> List[str]:
|
| 126 |
+
"""Get list of files that have changed since last scan.
|
| 127 |
+
|
| 128 |
+
Args:
|
| 129 |
+
file_paths: List of file paths to check.
|
| 130 |
+
|
| 131 |
+
Returns:
|
| 132 |
+
List of changed file paths.
|
| 133 |
+
"""
|
| 134 |
+
return [fp for fp in file_paths if self.is_file_changed(fp)]
|
| 135 |
+
|
| 136 |
+
def get_unchanged_files(self, file_paths: List[str]) -> List[str]:
|
| 137 |
+
"""Get list of files that haven't changed since last scan.
|
| 138 |
+
|
| 139 |
+
Args:
|
| 140 |
+
file_paths: List of file paths to check.
|
| 141 |
+
|
| 142 |
+
Returns:
|
| 143 |
+
List of unchanged file paths.
|
| 144 |
+
"""
|
| 145 |
+
return [fp for fp in file_paths if not self.is_file_changed(fp)]
|
| 146 |
+
|
| 147 |
+
def get_cached_findings(self, file_path: str) -> List[Dict[str, Any]]:
|
| 148 |
+
"""Get cached scan findings for a file.
|
| 149 |
+
|
| 150 |
+
Args:
|
| 151 |
+
file_path: Path to the file.
|
| 152 |
+
|
| 153 |
+
Returns:
|
| 154 |
+
List of cached findings, or empty list if not cached.
|
| 155 |
+
"""
|
| 156 |
+
self._ensure_loaded()
|
| 157 |
+
file_entry = self._cache.get("files", {}).get(file_path, {})
|
| 158 |
+
return file_entry.get("findings", [])
|
| 159 |
+
|
| 160 |
+
def update_file(self, file_path: str, findings: List[Dict[str, Any]]) -> None:
|
| 161 |
+
"""Update cache for a file with new findings.
|
| 162 |
+
|
| 163 |
+
Args:
|
| 164 |
+
file_path: Path to the file.
|
| 165 |
+
findings: List of findings for this file.
|
| 166 |
+
"""
|
| 167 |
+
self._ensure_loaded()
|
| 168 |
+
current_hash = self.compute_file_hash(file_path)
|
| 169 |
+
|
| 170 |
+
if "files" not in self._cache:
|
| 171 |
+
self._cache["files"] = {}
|
| 172 |
+
|
| 173 |
+
self._cache["files"][file_path] = {
|
| 174 |
+
"hash": current_hash,
|
| 175 |
+
"findings": findings,
|
| 176 |
+
"last_scanned": time.time(),
|
| 177 |
+
"finding_count": len(findings),
|
| 178 |
+
}
|
| 179 |
+
self.changed = True
|
| 180 |
+
|
| 181 |
+
def invalidate_file(self, file_path: str) -> None:
|
| 182 |
+
"""Remove a file from the cache.
|
| 183 |
+
|
| 184 |
+
Args:
|
| 185 |
+
file_path: Path to the file.
|
| 186 |
+
"""
|
| 187 |
+
self._ensure_loaded()
|
| 188 |
+
if file_path in self._cache.get("files", {}):
|
| 189 |
+
del self._cache["files"][file_path]
|
| 190 |
+
self.changed = True
|
| 191 |
+
|
| 192 |
+
def clear(self) -> None:
|
| 193 |
+
"""Clear the entire cache."""
|
| 194 |
+
self._cache = {"files": {}, "metadata": {}}
|
| 195 |
+
self.changed = True
|
| 196 |
+
|
| 197 |
+
def cleanup_expired(self) -> int:
|
| 198 |
+
"""Remove expired cache entries.
|
| 199 |
+
|
| 200 |
+
Returns:
|
| 201 |
+
Number of expired entries removed.
|
| 202 |
+
"""
|
| 203 |
+
self._ensure_loaded()
|
| 204 |
+
if not self.max_age_days:
|
| 205 |
+
return 0
|
| 206 |
+
|
| 207 |
+
cutoff = time.time() - (self.max_age_days * 86400)
|
| 208 |
+
expired = []
|
| 209 |
+
|
| 210 |
+
for file_path, entry in self._cache.get("files", {}).items():
|
| 211 |
+
if entry.get("last_scanned", 0) < cutoff:
|
| 212 |
+
expired.append(file_path)
|
| 213 |
+
|
| 214 |
+
for file_path in expired:
|
| 215 |
+
del self._cache["files"][file_path]
|
| 216 |
+
|
| 217 |
+
if expired:
|
| 218 |
+
self.changed = True
|
| 219 |
+
logger.info(f"Cleaned up {len(expired)} expired cache entries")
|
| 220 |
+
|
| 221 |
+
return len(expired)
|
| 222 |
+
|
| 223 |
+
def get_stats(self) -> Dict[str, Any]:
|
| 224 |
+
"""Get cache statistics.
|
| 225 |
+
|
| 226 |
+
Returns:
|
| 227 |
+
Dictionary with cache statistics.
|
| 228 |
+
"""
|
| 229 |
+
self._ensure_loaded()
|
| 230 |
+
files = self._cache.get("files", {})
|
| 231 |
+
total_findings = sum(e.get("finding_count", 0) for e in files.values())
|
| 232 |
+
|
| 233 |
+
return {
|
| 234 |
+
"cached_files": len(files),
|
| 235 |
+
"total_cached_findings": total_findings,
|
| 236 |
+
"cache_size_bytes": self.cache_file.stat().st_size if self.cache_file.exists() else 0,
|
| 237 |
+
"last_saved": self._cache.get("metadata", {}).get("last_saved"),
|
| 238 |
+
}
|
shield_agents/cli.py
ADDED
|
@@ -0,0 +1,420 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Command-line interface for Shield Agents.
|
| 3 |
+
|
| 4 |
+
Provides a beautiful, interactive CLI using the Rich library for
|
| 5 |
+
progress bars, tables, and colored output.
|
| 6 |
+
"""
|
| 7 |
+
|
| 8 |
+
import argparse
|
| 9 |
+
import asyncio
|
| 10 |
+
import json
|
| 11 |
+
import logging
|
| 12 |
+
import sys
|
| 13 |
+
from pathlib import Path
|
| 14 |
+
from typing import Optional
|
| 15 |
+
|
| 16 |
+
from . import __version__
|
| 17 |
+
from .config import ShieldConfig
|
| 18 |
+
from .orchestrator import Orchestrator
|
| 19 |
+
from .shieldignore import ShieldIgnore
|
| 20 |
+
from .cache import ScanCache
|
| 21 |
+
|
| 22 |
+
logger = logging.getLogger("shield_agents.cli")
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def setup_logging(verbose: bool = False, debug: bool = False):
|
| 26 |
+
"""Configure logging based on verbosity."""
|
| 27 |
+
level = logging.DEBUG if debug else (logging.INFO if verbose else logging.WARNING)
|
| 28 |
+
logging.basicConfig(
|
| 29 |
+
level=level,
|
| 30 |
+
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
|
| 31 |
+
datefmt="%H:%M:%S",
|
| 32 |
+
)
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def create_parser() -> argparse.ArgumentParser:
|
| 36 |
+
"""Create the CLI argument parser."""
|
| 37 |
+
parser = argparse.ArgumentParser(
|
| 38 |
+
prog="shield-agents",
|
| 39 |
+
description="Shield Agents - AI-Powered Multi-Agent Cybersecurity Scanner",
|
| 40 |
+
formatter_class=argparse.RawDescriptionHelpFormatter,
|
| 41 |
+
epilog="""
|
| 42 |
+
Examples:
|
| 43 |
+
shield-agents scan ./my-project # Scan a project
|
| 44 |
+
shield-agents scan ./my-project --full # Full scan (ignore cache)
|
| 45 |
+
shield-agents scan ./my-project --fix # Scan with auto-fix suggestions
|
| 46 |
+
shield-agents scan ./my-project --sarif-only # Output only SARIF for GitHub
|
| 47 |
+
shield-agents init # Create .shieldignore template
|
| 48 |
+
shield-agents cache --clear # Clear scan cache
|
| 49 |
+
shield-agents cache --stats # Show cache statistics
|
| 50 |
+
shield-agents version # Show version info
|
| 51 |
+
""",
|
| 52 |
+
)
|
| 53 |
+
|
| 54 |
+
subparsers = parser.add_subparsers(dest="command", help="Available commands")
|
| 55 |
+
|
| 56 |
+
# Scan command
|
| 57 |
+
scan_parser = subparsers.add_parser("scan", help="Run security scan")
|
| 58 |
+
scan_parser.add_argument("target", help="Path to scan")
|
| 59 |
+
scan_parser.add_argument("--config", "-c", help="Path to config YAML file")
|
| 60 |
+
scan_parser.add_argument("--full", action="store_true", help="Full scan (ignore cache)")
|
| 61 |
+
scan_parser.add_argument("--fix", action="store_true", help="Generate auto-fix suggestions")
|
| 62 |
+
scan_parser.add_argument("--no-report", action="store_true", help="Skip report generation")
|
| 63 |
+
scan_parser.add_argument("--sarif-only", action="store_true", help="Output only SARIF format")
|
| 64 |
+
scan_parser.add_argument("--output", "-o", help="Output directory for reports")
|
| 65 |
+
scan_parser.add_argument("--provider", choices=["mock", "openai", "anthropic", "ollama"], help="LLM provider")
|
| 66 |
+
scan_parser.add_argument("--verbose", "-v", action="store_true", help="Verbose output")
|
| 67 |
+
scan_parser.add_argument("--debug", action="store_true", help="Debug output")
|
| 68 |
+
scan_parser.add_argument("--no-cache", action="store_true", help="Disable caching")
|
| 69 |
+
scan_parser.add_argument("--no-dedup", action="store_true", help="Disable deduplication")
|
| 70 |
+
scan_parser.add_argument("--no-ignore", action="store_true", help="Ignore .shieldignore rules")
|
| 71 |
+
scan_parser.add_argument("--ci", action="store_true", help="CI/CD mode: SARIF output, JSON to stdout, silent except errors")
|
| 72 |
+
scan_parser.add_argument("--format", "-f", choices=["rich", "json", "sarif", "plain"], default="rich", help="Output format (default: rich)")
|
| 73 |
+
scan_parser.add_argument("--fail-threshold", type=int, default=75, help="Risk score threshold for CI failure (default: 75)")
|
| 74 |
+
|
| 75 |
+
# Init command
|
| 76 |
+
init_parser = subparsers.add_parser("init", help="Initialize Shield Agents configuration")
|
| 77 |
+
init_parser.add_argument("--path", default=".", help="Directory to initialize")
|
| 78 |
+
|
| 79 |
+
# Cache command
|
| 80 |
+
cache_parser = subparsers.add_parser("cache", help="Manage scan cache")
|
| 81 |
+
cache_parser.add_argument("--clear", action="store_true", help="Clear the cache")
|
| 82 |
+
cache_parser.add_argument("--stats", action="store_true", help="Show cache statistics")
|
| 83 |
+
|
| 84 |
+
# Version command
|
| 85 |
+
subparsers.add_parser("version", help="Show version information")
|
| 86 |
+
|
| 87 |
+
return parser
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
async def run_scan_ci(args) -> int:
|
| 91 |
+
"""CI/CD optimized scan - silent except for JSON output on stdout."""
|
| 92 |
+
config = ShieldConfig(config_path=args.config) if args.config else ShieldConfig()
|
| 93 |
+
config.target_path = args.target
|
| 94 |
+
|
| 95 |
+
# Apply CLI overrides
|
| 96 |
+
if args.provider:
|
| 97 |
+
config.llm.provider = args.provider
|
| 98 |
+
if args.output:
|
| 99 |
+
config.report.output_dir = args.output
|
| 100 |
+
if args.no_cache:
|
| 101 |
+
config.cache.enabled = False
|
| 102 |
+
if args.no_dedup:
|
| 103 |
+
config.deduplication.enabled = False
|
| 104 |
+
|
| 105 |
+
# CI mode: always generate SARIF + JSON (or SARIF only if --sarif-only)
|
| 106 |
+
config.report.formats = ["sarif", "json"] if not getattr(args, 'sarif_only', False) else ["sarif"]
|
| 107 |
+
|
| 108 |
+
# Minimal logging for CI
|
| 109 |
+
setup_logging(verbose=False, debug=False)
|
| 110 |
+
|
| 111 |
+
orchestrator = Orchestrator(config)
|
| 112 |
+
result = await orchestrator.scan(
|
| 113 |
+
target_path=args.target,
|
| 114 |
+
full_scan=args.full,
|
| 115 |
+
generate_report=True,
|
| 116 |
+
generate_fixes=args.fix,
|
| 117 |
+
)
|
| 118 |
+
|
| 119 |
+
# Output JSON to stdout for CI pipeline consumption
|
| 120 |
+
output = result.to_dict()
|
| 121 |
+
print(json.dumps(output, indent=2))
|
| 122 |
+
|
| 123 |
+
# Exit code based on risk threshold
|
| 124 |
+
threshold = getattr(args, 'fail_threshold', 75) or 75
|
| 125 |
+
return 1 if result.risk_score >= threshold else 0
|
| 126 |
+
|
| 127 |
+
|
| 128 |
+
async def run_scan(args) -> int:
|
| 129 |
+
"""Execute the scan command."""
|
| 130 |
+
# If format is json, use CI-style output
|
| 131 |
+
if getattr(args, 'format', 'rich') == 'json':
|
| 132 |
+
return await run_scan_ci(args)
|
| 133 |
+
|
| 134 |
+
try:
|
| 135 |
+
from rich.console import Console
|
| 136 |
+
from rich.progress import Progress, SpinnerColumn, TextColumn
|
| 137 |
+
from rich.table import Table
|
| 138 |
+
from rich.panel import Panel
|
| 139 |
+
from rich import box
|
| 140 |
+
except ImportError:
|
| 141 |
+
# Fallback without Rich
|
| 142 |
+
return await _run_scan_simple(args)
|
| 143 |
+
|
| 144 |
+
console = Console()
|
| 145 |
+
|
| 146 |
+
# Load configuration
|
| 147 |
+
config = ShieldConfig(config_path=args.config) if args.config else ShieldConfig()
|
| 148 |
+
config.target_path = args.target
|
| 149 |
+
|
| 150 |
+
# Apply CLI overrides
|
| 151 |
+
if args.provider:
|
| 152 |
+
config.llm.provider = args.provider
|
| 153 |
+
if args.output:
|
| 154 |
+
config.report.output_dir = args.output
|
| 155 |
+
if args.no_cache:
|
| 156 |
+
config.cache.enabled = False
|
| 157 |
+
if args.no_dedup:
|
| 158 |
+
config.deduplication.enabled = False
|
| 159 |
+
if args.no_ignore:
|
| 160 |
+
config.shieldignore.enabled = False
|
| 161 |
+
if args.sarif_only:
|
| 162 |
+
config.report.formats = ["sarif"]
|
| 163 |
+
if args.ci:
|
| 164 |
+
config.report.formats = ["sarif", "json"]
|
| 165 |
+
|
| 166 |
+
config.verbose = args.verbose
|
| 167 |
+
config.debug = args.debug
|
| 168 |
+
setup_logging(args.verbose, args.debug)
|
| 169 |
+
|
| 170 |
+
# Display header
|
| 171 |
+
console.print(Panel.fit(
|
| 172 |
+
"[bold blue]Shield Agents[/bold blue] - AI-Powered Security Scanner\n"
|
| 173 |
+
f"[dim]Version {__version__} | Provider: {config.llm.provider}[/dim]",
|
| 174 |
+
border_style="blue",
|
| 175 |
+
))
|
| 176 |
+
|
| 177 |
+
# Run scan
|
| 178 |
+
orchestrator = Orchestrator(config)
|
| 179 |
+
|
| 180 |
+
with Progress(
|
| 181 |
+
SpinnerColumn(),
|
| 182 |
+
TextColumn("[progress.description]{task.description}"),
|
| 183 |
+
console=console,
|
| 184 |
+
) as progress:
|
| 185 |
+
task = progress.add_task("Scanning...", total=None)
|
| 186 |
+
result = await orchestrator.scan(
|
| 187 |
+
target_path=args.target,
|
| 188 |
+
full_scan=args.full,
|
| 189 |
+
generate_report=not args.no_report,
|
| 190 |
+
generate_fixes=args.fix,
|
| 191 |
+
)
|
| 192 |
+
progress.update(task, completed=True)
|
| 193 |
+
|
| 194 |
+
# Display results
|
| 195 |
+
severity_colors = {
|
| 196 |
+
"CRITICAL": "bold red",
|
| 197 |
+
"HIGH": "red",
|
| 198 |
+
"MEDIUM": "yellow",
|
| 199 |
+
"LOW": "cyan",
|
| 200 |
+
"INFO": "dim",
|
| 201 |
+
}
|
| 202 |
+
|
| 203 |
+
# Risk score panel
|
| 204 |
+
risk_color = "red" if result.risk_score >= 75 else "yellow" if result.risk_score >= 50 else "green"
|
| 205 |
+
console.print(Panel.fit(
|
| 206 |
+
f"[{risk_color}]Risk Score: {result.risk_score}/100[/{risk_color}]\n"
|
| 207 |
+
f"Files scanned: {result.files_scanned}\n"
|
| 208 |
+
f"Total findings: {len(result.filtered_findings)}\n"
|
| 209 |
+
f"Scan duration: {result.scan_duration:.2f}s",
|
| 210 |
+
title="Scan Summary",
|
| 211 |
+
))
|
| 212 |
+
|
| 213 |
+
# Severity breakdown table
|
| 214 |
+
sev_table = Table(title="Severity Breakdown", box=box.SIMPLE)
|
| 215 |
+
sev_table.add_column("Severity", style="bold")
|
| 216 |
+
sev_table.add_column("Count", justify="right")
|
| 217 |
+
|
| 218 |
+
severity_counts = {}
|
| 219 |
+
for f in result.filtered_findings:
|
| 220 |
+
sev = f.get("severity", "MEDIUM").upper()
|
| 221 |
+
severity_counts[sev] = severity_counts.get(sev, 0) + 1
|
| 222 |
+
|
| 223 |
+
for sev in ["CRITICAL", "HIGH", "MEDIUM", "LOW", "INFO"]:
|
| 224 |
+
count = severity_counts.get(sev, 0)
|
| 225 |
+
if count > 0:
|
| 226 |
+
sev_table.add_row(f"[{severity_colors.get(sev, 'white')}]{sev}[/{severity_colors.get(sev, 'white')}]", str(count))
|
| 227 |
+
|
| 228 |
+
console.print(sev_table)
|
| 229 |
+
|
| 230 |
+
# Findings table
|
| 231 |
+
if result.filtered_findings:
|
| 232 |
+
findings_table = Table(title="Findings", box=box.SIMPLE, show_lines=True)
|
| 233 |
+
findings_table.add_column("#", style="dim", width=4)
|
| 234 |
+
findings_table.add_column("Severity", width=10)
|
| 235 |
+
findings_table.add_column("Title", width=40)
|
| 236 |
+
findings_table.add_column("File", width=30)
|
| 237 |
+
findings_table.add_column("Line", width=6)
|
| 238 |
+
findings_table.add_column("Source", width=15)
|
| 239 |
+
|
| 240 |
+
for i, f in enumerate(result.filtered_findings[:50], 1):
|
| 241 |
+
sev = f.get("severity", "MEDIUM").upper()
|
| 242 |
+
sev_style = severity_colors.get(sev, "white")
|
| 243 |
+
findings_table.add_row(
|
| 244 |
+
str(i),
|
| 245 |
+
f"[{sev_style}]{sev}[/{sev_style}]",
|
| 246 |
+
f.get("title", "Unknown")[:40],
|
| 247 |
+
f.get("file", "N/A")[-30:],
|
| 248 |
+
str(f.get("line", "N/A")),
|
| 249 |
+
f.get("source", f.get("agent", "unknown"))[:15],
|
| 250 |
+
)
|
| 251 |
+
|
| 252 |
+
console.print(findings_table)
|
| 253 |
+
|
| 254 |
+
# Auto-fix suggestions
|
| 255 |
+
if result.fixes:
|
| 256 |
+
console.print(f"\n[bold green]Auto-fix Suggestions ({len(result.fixes)}):[/bold green]")
|
| 257 |
+
for i, fix in enumerate(result.fixes[:20], 1):
|
| 258 |
+
console.print(f" {i}. [{severity_colors.get(fix.get('severity', 'INFO'), 'dim')}]{fix.get('title', 'Unknown')}[/{severity_colors.get(fix.get('severity', 'INFO'), 'dim')}]")
|
| 259 |
+
if fix.get("fix"):
|
| 260 |
+
console.print(f" [dim]Fix: {fix['fix'][:100]}[/dim]")
|
| 261 |
+
|
| 262 |
+
# Report locations
|
| 263 |
+
if result.report_files:
|
| 264 |
+
console.print("\n[bold]Reports generated:[/bold]")
|
| 265 |
+
for fmt, path in result.report_files.items():
|
| 266 |
+
console.print(f" [{fmt}] {path}")
|
| 267 |
+
|
| 268 |
+
# Stats
|
| 269 |
+
if result.stats:
|
| 270 |
+
console.print(f"\n[dim]Stats: {result.stats}[/dim]")
|
| 271 |
+
|
| 272 |
+
return 0 if result.risk_score < 75 else 1
|
| 273 |
+
|
| 274 |
+
|
| 275 |
+
async def _run_scan_simple(args) -> int:
|
| 276 |
+
"""Simple scan output without Rich library."""
|
| 277 |
+
config = ShieldConfig(config_path=args.config) if args.config else ShieldConfig()
|
| 278 |
+
config.target_path = args.target
|
| 279 |
+
|
| 280 |
+
if args.provider:
|
| 281 |
+
config.llm.provider = args.provider
|
| 282 |
+
if args.output:
|
| 283 |
+
config.report.output_dir = args.output
|
| 284 |
+
if args.no_cache:
|
| 285 |
+
config.cache.enabled = False
|
| 286 |
+
if args.no_dedup:
|
| 287 |
+
config.deduplication.enabled = False
|
| 288 |
+
if args.no_ignore:
|
| 289 |
+
config.shieldignore.enabled = False
|
| 290 |
+
if args.sarif_only:
|
| 291 |
+
config.report.formats = ["sarif"]
|
| 292 |
+
if args.ci:
|
| 293 |
+
config.report.formats = ["sarif", "json"]
|
| 294 |
+
|
| 295 |
+
setup_logging(args.verbose, args.debug)
|
| 296 |
+
|
| 297 |
+
print(f"Shield Agents v{__version__} - Scanning {args.target}...")
|
| 298 |
+
orchestrator = Orchestrator(config)
|
| 299 |
+
result = await orchestrator.scan(
|
| 300 |
+
target_path=args.target,
|
| 301 |
+
full_scan=args.full,
|
| 302 |
+
generate_report=not args.no_report,
|
| 303 |
+
generate_fixes=args.fix,
|
| 304 |
+
)
|
| 305 |
+
|
| 306 |
+
# JSON format: output to stdout and exit
|
| 307 |
+
if args.format == "json":
|
| 308 |
+
output = result.to_dict()
|
| 309 |
+
print(json.dumps(output, indent=2))
|
| 310 |
+
threshold = getattr(args, 'fail_threshold', 75) or 75
|
| 311 |
+
return 1 if result.risk_score >= threshold else 0
|
| 312 |
+
|
| 313 |
+
print(f"\nRisk Score: {result.risk_score}/100")
|
| 314 |
+
print(f"Files scanned: {result.files_scanned}")
|
| 315 |
+
print(f"Findings: {len(result.filtered_findings)}")
|
| 316 |
+
print(f"Duration: {result.scan_duration:.2f}s")
|
| 317 |
+
|
| 318 |
+
if result.report_files:
|
| 319 |
+
print("\nReports:")
|
| 320 |
+
for fmt, path in result.report_files.items():
|
| 321 |
+
print(f" [{fmt}] {path}")
|
| 322 |
+
|
| 323 |
+
return 0 if result.risk_score < 75 else 1
|
| 324 |
+
|
| 325 |
+
|
| 326 |
+
def run_init(args) -> int:
|
| 327 |
+
"""Initialize Shield Agents configuration."""
|
| 328 |
+
path = args.path
|
| 329 |
+
template_path = ShieldIgnore.create_template(path)
|
| 330 |
+
print(f"Created .shieldignore template at: {template_path}")
|
| 331 |
+
|
| 332 |
+
# Also create a default config
|
| 333 |
+
config_path = Path(path) / "config.yaml"
|
| 334 |
+
if not config_path.exists():
|
| 335 |
+
config_content = """# Shield Agents Configuration
|
| 336 |
+
llm:
|
| 337 |
+
provider: mock # mock, openai, anthropic, ollama
|
| 338 |
+
# api_key: YOUR_API_KEY # Or set SHIELD_LLM_API_KEY env var
|
| 339 |
+
model: gpt-4
|
| 340 |
+
temperature: 0.1
|
| 341 |
+
|
| 342 |
+
scanner:
|
| 343 |
+
sast_enabled: true
|
| 344 |
+
secrets_enabled: true
|
| 345 |
+
|
| 346 |
+
cache:
|
| 347 |
+
enabled: true
|
| 348 |
+
incremental: true
|
| 349 |
+
|
| 350 |
+
deduplication:
|
| 351 |
+
enabled: true
|
| 352 |
+
merge_sources: true
|
| 353 |
+
|
| 354 |
+
report:
|
| 355 |
+
output_dir: ./shield-reports
|
| 356 |
+
formats:
|
| 357 |
+
- html
|
| 358 |
+
- sarif
|
| 359 |
+
- json
|
| 360 |
+
"""
|
| 361 |
+
Path(path).mkdir(parents=True, exist_ok=True)
|
| 362 |
+
config_path.write_text(config_content)
|
| 363 |
+
print(f"Created config template at: {config_path}")
|
| 364 |
+
|
| 365 |
+
return 0
|
| 366 |
+
|
| 367 |
+
|
| 368 |
+
def run_cache(args) -> int:
|
| 369 |
+
"""Manage scan cache."""
|
| 370 |
+
cache = ScanCache()
|
| 371 |
+
cache._ensure_loaded()
|
| 372 |
+
|
| 373 |
+
if args.clear:
|
| 374 |
+
cache.clear()
|
| 375 |
+
cache.save()
|
| 376 |
+
print("Cache cleared.")
|
| 377 |
+
elif args.stats:
|
| 378 |
+
stats = cache.get_stats()
|
| 379 |
+
print(f"Cached files: {stats['cached_files']}")
|
| 380 |
+
print(f"Total cached findings: {stats['total_cached_findings']}")
|
| 381 |
+
print(f"Cache size: {stats['cache_size_bytes']} bytes")
|
| 382 |
+
else:
|
| 383 |
+
print("Use --clear or --stats")
|
| 384 |
+
|
| 385 |
+
return 0
|
| 386 |
+
|
| 387 |
+
|
| 388 |
+
def main():
|
| 389 |
+
"""Main entry point."""
|
| 390 |
+
parser = create_parser()
|
| 391 |
+
args = parser.parse_args()
|
| 392 |
+
|
| 393 |
+
if args.command is None:
|
| 394 |
+
parser.print_help()
|
| 395 |
+
return 0
|
| 396 |
+
|
| 397 |
+
if args.command == "version":
|
| 398 |
+
print(f"Shield Agents v{__version__}")
|
| 399 |
+
return 0
|
| 400 |
+
|
| 401 |
+
if args.command == "init":
|
| 402 |
+
return run_init(args)
|
| 403 |
+
|
| 404 |
+
if args.command == "cache":
|
| 405 |
+
return run_cache(args)
|
| 406 |
+
|
| 407 |
+
if args.command == "scan":
|
| 408 |
+
try:
|
| 409 |
+
if args.ci:
|
| 410 |
+
return asyncio.run(run_scan_ci(args))
|
| 411 |
+
return asyncio.run(run_scan(args))
|
| 412 |
+
except KeyboardInterrupt:
|
| 413 |
+
print("\nScan interrupted.")
|
| 414 |
+
return 130
|
| 415 |
+
|
| 416 |
+
return 0
|
| 417 |
+
|
| 418 |
+
|
| 419 |
+
if __name__ == "__main__":
|
| 420 |
+
sys.exit(main())
|
shield_agents/config.py
ADDED
|
@@ -0,0 +1,209 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Configuration management for Shield Agents.
|
| 3 |
+
|
| 4 |
+
Supports YAML config files, environment variables, and sensible defaults.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
import os
|
| 8 |
+
import yaml
|
| 9 |
+
from dataclasses import dataclass, field
|
| 10 |
+
from typing import Dict, List, Optional
|
| 11 |
+
from pathlib import Path
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
@dataclass
|
| 15 |
+
class LLMConfig:
|
| 16 |
+
"""LLM provider configuration."""
|
| 17 |
+
provider: str = "mock" # mock, openai, anthropic, ollama
|
| 18 |
+
api_key: Optional[str] = None
|
| 19 |
+
model: str = "gpt-4"
|
| 20 |
+
base_url: Optional[str] = None
|
| 21 |
+
temperature: float = 0.1
|
| 22 |
+
max_tokens: int = 4096
|
| 23 |
+
timeout: int = 60
|
| 24 |
+
max_retries: int = 3
|
| 25 |
+
fallback_enabled: bool = True # Enable robust fallback parsing
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
@dataclass
|
| 29 |
+
class ScannerConfig:
|
| 30 |
+
"""Scanner-specific configuration."""
|
| 31 |
+
sast_enabled: bool = True
|
| 32 |
+
secrets_enabled: bool = True
|
| 33 |
+
dependency_enabled: bool = True
|
| 34 |
+
max_file_size: int = 1024 * 1024 # 1MB
|
| 35 |
+
excluded_extensions: List[str] = field(default_factory=lambda: [
|
| 36 |
+
".png", ".jpg", ".jpeg", ".gif", ".bmp", ".ico", ".svg",
|
| 37 |
+
".mp3", ".mp4", ".avi", ".mov", ".wav",
|
| 38 |
+
".zip", ".tar", ".gz", ".rar", ".7z",
|
| 39 |
+
".pdf", ".doc", ".docx", ".xls", ".xlsx",
|
| 40 |
+
".woff", ".woff2", ".ttf", ".eot",
|
| 41 |
+
])
|
| 42 |
+
excluded_dirs: List[str] = field(default_factory=lambda: [
|
| 43 |
+
"node_modules", ".git", "__pycache__", ".venv", "venv",
|
| 44 |
+
"dist", "build", ".tox", ".mypy_cache", ".pytest_cache",
|
| 45 |
+
"shield-cache", ".shield-cache", "shield-reports",
|
| 46 |
+
])
|
| 47 |
+
excluded_content_dirs: List[str] = field(default_factory=lambda: [
|
| 48 |
+
"tests", "test", "tests_", "__tests__",
|
| 49 |
+
"benchmarks", "benchmark",
|
| 50 |
+
"examples", "example", "examples_",
|
| 51 |
+
"fixtures", "mocks",
|
| 52 |
+
])
|
| 53 |
+
excluded_filenames: List[str] = field(default_factory=lambda: [
|
| 54 |
+
"setup.py", "conftest.py",
|
| 55 |
+
])
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
@dataclass
|
| 59 |
+
class CacheConfig:
|
| 60 |
+
"""Caching and incremental scan configuration."""
|
| 61 |
+
enabled: bool = True
|
| 62 |
+
cache_dir: str = ".shield-cache"
|
| 63 |
+
hash_algorithm: str = "sha256"
|
| 64 |
+
max_cache_age_days: int = 30
|
| 65 |
+
incremental: bool = True
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
@dataclass
|
| 69 |
+
class DeduplicationConfig:
|
| 70 |
+
"""Deduplication engine configuration."""
|
| 71 |
+
enabled: bool = True
|
| 72 |
+
similarity_threshold: float = 0.85
|
| 73 |
+
merge_sources: bool = True
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
@dataclass
|
| 77 |
+
class SARIFConfig:
|
| 78 |
+
"""SARIF output configuration."""
|
| 79 |
+
enabled: bool = True
|
| 80 |
+
schema_uri: str = "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json"
|
| 81 |
+
version: str = "2.1.0"
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
@dataclass
|
| 85 |
+
class ShieldIgnoreConfig:
|
| 86 |
+
"""Shieldignore file configuration."""
|
| 87 |
+
enabled: bool = True
|
| 88 |
+
filename: str = ".shieldignore"
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
@dataclass
|
| 92 |
+
class AgentConfig:
|
| 93 |
+
"""Individual agent configuration."""
|
| 94 |
+
vuln_agent: bool = True
|
| 95 |
+
threat_agent: bool = True
|
| 96 |
+
recon_agent: bool = True
|
| 97 |
+
compliance_agent: bool = True
|
| 98 |
+
response_agent: bool = True
|
| 99 |
+
autofix_agent: bool = True
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
@dataclass
|
| 103 |
+
class ReportConfig:
|
| 104 |
+
"""Report generation configuration."""
|
| 105 |
+
output_dir: str = "./shield-reports"
|
| 106 |
+
formats: List[str] = field(default_factory=lambda: ["html", "sarif", "json"])
|
| 107 |
+
include_remediation: bool = True
|
| 108 |
+
include_code_snippets: bool = True
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
class ShieldConfig:
|
| 112 |
+
"""Main configuration class that aggregates all sub-configs."""
|
| 113 |
+
|
| 114 |
+
def __init__(self, config_path: Optional[str] = None):
|
| 115 |
+
self.llm = LLMConfig()
|
| 116 |
+
self.scanner = ScannerConfig()
|
| 117 |
+
self.cache = CacheConfig()
|
| 118 |
+
self.deduplication = DeduplicationConfig()
|
| 119 |
+
self.sarif = SARIFConfig()
|
| 120 |
+
self.shieldignore = ShieldIgnoreConfig()
|
| 121 |
+
self.agents = AgentConfig()
|
| 122 |
+
self.report = ReportConfig()
|
| 123 |
+
self.target_path: str = "."
|
| 124 |
+
self.verbose: bool = False
|
| 125 |
+
self.debug: bool = False
|
| 126 |
+
|
| 127 |
+
# Load from environment variables first
|
| 128 |
+
self._load_env()
|
| 129 |
+
|
| 130 |
+
# Then override with config file if provided
|
| 131 |
+
if config_path:
|
| 132 |
+
self._load_yaml(config_path)
|
| 133 |
+
|
| 134 |
+
def _load_env(self):
|
| 135 |
+
"""Load configuration from environment variables."""
|
| 136 |
+
self.llm.provider = os.getenv("SHIELD_LLM_PROVIDER", self.llm.provider)
|
| 137 |
+
self.llm.api_key = os.getenv("SHIELD_LLM_API_KEY", self.llm.api_key)
|
| 138 |
+
self.llm.model = os.getenv("SHIELD_LLM_MODEL", self.llm.model)
|
| 139 |
+
self.llm.base_url = os.getenv("SHIELD_LLM_BASE_URL", self.llm.base_url)
|
| 140 |
+
|
| 141 |
+
if os.getenv("SHIELD_VERBOSE", "").lower() in ("1", "true", "yes"):
|
| 142 |
+
self.verbose = True
|
| 143 |
+
if os.getenv("SHIELD_DEBUG", "").lower() in ("1", "true", "yes"):
|
| 144 |
+
self.debug = True
|
| 145 |
+
|
| 146 |
+
def _load_yaml(self, config_path: str):
|
| 147 |
+
"""Load configuration from YAML file."""
|
| 148 |
+
path = Path(config_path)
|
| 149 |
+
if not path.exists():
|
| 150 |
+
return
|
| 151 |
+
|
| 152 |
+
with open(path, "r") as f:
|
| 153 |
+
data = yaml.safe_load(f) or {}
|
| 154 |
+
|
| 155 |
+
if "llm" in data:
|
| 156 |
+
for k, v in data["llm"].items():
|
| 157 |
+
if hasattr(self.llm, k):
|
| 158 |
+
setattr(self.llm, k, v)
|
| 159 |
+
|
| 160 |
+
if "scanner" in data:
|
| 161 |
+
for k, v in data["scanner"].items():
|
| 162 |
+
if hasattr(self.scanner, k):
|
| 163 |
+
setattr(self.scanner, k, v)
|
| 164 |
+
|
| 165 |
+
if "cache" in data:
|
| 166 |
+
for k, v in data["cache"].items():
|
| 167 |
+
if hasattr(self.cache, k):
|
| 168 |
+
setattr(self.cache, k, v)
|
| 169 |
+
|
| 170 |
+
if "deduplication" in data:
|
| 171 |
+
for k, v in data["deduplication"].items():
|
| 172 |
+
if hasattr(self.deduplication, k):
|
| 173 |
+
setattr(self.deduplication, k, v)
|
| 174 |
+
|
| 175 |
+
if "sarif" in data:
|
| 176 |
+
for k, v in data["sarif"].items():
|
| 177 |
+
if hasattr(self.sarif, k):
|
| 178 |
+
setattr(self.sarif, k, v)
|
| 179 |
+
|
| 180 |
+
if "agents" in data:
|
| 181 |
+
for k, v in data["agents"].items():
|
| 182 |
+
if hasattr(self.agents, k):
|
| 183 |
+
setattr(self.agents, k, v)
|
| 184 |
+
|
| 185 |
+
if "report" in data:
|
| 186 |
+
for k, v in data["report"].items():
|
| 187 |
+
if hasattr(self.report, k):
|
| 188 |
+
setattr(self.report, k, v)
|
| 189 |
+
|
| 190 |
+
if "target_path" in data:
|
| 191 |
+
self.target_path = data["target_path"]
|
| 192 |
+
|
| 193 |
+
@classmethod
|
| 194 |
+
def from_file(cls, config_path: str) -> "ShieldConfig":
|
| 195 |
+
"""Create configuration from a YAML file."""
|
| 196 |
+
return cls(config_path=config_path)
|
| 197 |
+
|
| 198 |
+
def to_dict(self) -> dict:
|
| 199 |
+
"""Export configuration as dictionary."""
|
| 200 |
+
import dataclasses
|
| 201 |
+
result = {}
|
| 202 |
+
for attr_name in ["llm", "scanner", "cache", "deduplication", "sarif", "shieldignore", "agents", "report"]:
|
| 203 |
+
attr = getattr(self, attr_name)
|
| 204 |
+
if dataclasses.is_dataclass(attr):
|
| 205 |
+
result[attr_name] = dataclasses.asdict(attr)
|
| 206 |
+
result["target_path"] = self.target_path
|
| 207 |
+
result["verbose"] = self.verbose
|
| 208 |
+
result["debug"] = self.debug
|
| 209 |
+
return result
|
shield_agents/deduplication.py
ADDED
|
@@ -0,0 +1,254 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Deduplication Engine for Shield Agents.
|
| 3 |
+
|
| 4 |
+
When VulnAgent finds "SQL Injection" and SAST also finds "SQL Injection",
|
| 5 |
+
this engine merges them into one finding with multiple sources. This prevents
|
| 6 |
+
duplicate findings from cluttering reports and inflating risk scores.
|
| 7 |
+
"""
|
| 8 |
+
|
| 9 |
+
import logging
|
| 10 |
+
import re
|
| 11 |
+
from difflib import SequenceMatcher
|
| 12 |
+
from typing import Any, Dict, List, Optional, Set, Tuple
|
| 13 |
+
|
| 14 |
+
logger = logging.getLogger("shield_agents.deduplication")
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
class FindingDeduplicator:
|
| 18 |
+
"""Deduplicate findings across multiple agents and scanners.
|
| 19 |
+
|
| 20 |
+
Uses multiple strategies:
|
| 21 |
+
1. Exact match: Same file, line, and category
|
| 22 |
+
2. Fuzzy match: Similar title/description within same file
|
| 23 |
+
3. Category merge: Same vulnerability type at nearby lines
|
| 24 |
+
"""
|
| 25 |
+
|
| 26 |
+
def __init__(self, similarity_threshold: float = 0.85, merge_sources: bool = True):
|
| 27 |
+
"""Initialize the deduplicator.
|
| 28 |
+
|
| 29 |
+
Args:
|
| 30 |
+
similarity_threshold: Threshold for fuzzy matching (0.0-1.0).
|
| 31 |
+
merge_sources: Whether to merge sources into a single finding.
|
| 32 |
+
"""
|
| 33 |
+
self.similarity_threshold = similarity_threshold
|
| 34 |
+
self.merge_sources = merge_sources
|
| 35 |
+
self.dedup_stats = {
|
| 36 |
+
"total_input": 0,
|
| 37 |
+
"exact_duplicates": 0,
|
| 38 |
+
"fuzzy_duplicates": 0,
|
| 39 |
+
"category_merges": 0,
|
| 40 |
+
"final_count": 0,
|
| 41 |
+
}
|
| 42 |
+
|
| 43 |
+
def deduplicate(self, findings: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
| 44 |
+
"""Deduplicate a list of findings.
|
| 45 |
+
|
| 46 |
+
Args:
|
| 47 |
+
findings: List of finding dictionaries from multiple sources.
|
| 48 |
+
|
| 49 |
+
Returns:
|
| 50 |
+
Deduplicated list of findings.
|
| 51 |
+
"""
|
| 52 |
+
if not findings:
|
| 53 |
+
return []
|
| 54 |
+
|
| 55 |
+
self.dedup_stats["total_input"] = len(findings)
|
| 56 |
+
result = []
|
| 57 |
+
seen_keys: Set[str] = set()
|
| 58 |
+
merged_findings: Dict[str, Dict[str, Any]] = {}
|
| 59 |
+
|
| 60 |
+
# Phase 1: Exact deduplication
|
| 61 |
+
for finding in findings:
|
| 62 |
+
key = self._exact_key(finding)
|
| 63 |
+
if key in seen_keys:
|
| 64 |
+
self.dedup_stats["exact_duplicates"] += 1
|
| 65 |
+
# Merge sources into existing finding
|
| 66 |
+
if self.merge_sources and key in merged_findings:
|
| 67 |
+
self._merge_into(merged_findings[key], finding)
|
| 68 |
+
continue
|
| 69 |
+
|
| 70 |
+
seen_keys.add(key)
|
| 71 |
+
merged_findings[key] = {**finding}
|
| 72 |
+
if "sources" not in merged_findings[key]:
|
| 73 |
+
merged_findings[key]["sources"] = [finding.get("source", finding.get("agent", "unknown"))]
|
| 74 |
+
|
| 75 |
+
# Phase 2: Fuzzy deduplication (nearby lines, similar titles)
|
| 76 |
+
keys_list = list(merged_findings.keys())
|
| 77 |
+
to_remove: Set[str] = set()
|
| 78 |
+
|
| 79 |
+
for i, key1 in enumerate(keys_list):
|
| 80 |
+
if key1 in to_remove:
|
| 81 |
+
continue
|
| 82 |
+
for j in range(i + 1, len(keys_list)):
|
| 83 |
+
key2 = keys_list[j]
|
| 84 |
+
if key2 in to_remove:
|
| 85 |
+
continue
|
| 86 |
+
|
| 87 |
+
f1 = merged_findings[key1]
|
| 88 |
+
f2 = merged_findings[key2]
|
| 89 |
+
|
| 90 |
+
if self._is_fuzzy_duplicate(f1, f2):
|
| 91 |
+
self.dedup_stats["fuzzy_duplicates"] += 1
|
| 92 |
+
self._merge_into(f1, f2)
|
| 93 |
+
to_remove.add(key2)
|
| 94 |
+
|
| 95 |
+
# Phase 3: Category merge (same vuln type within 3 lines)
|
| 96 |
+
for i, key1 in enumerate(keys_list):
|
| 97 |
+
if key1 in to_remove:
|
| 98 |
+
continue
|
| 99 |
+
for j in range(i + 1, len(keys_list)):
|
| 100 |
+
key2 = keys_list[j]
|
| 101 |
+
if key2 in to_remove:
|
| 102 |
+
continue
|
| 103 |
+
|
| 104 |
+
f1 = merged_findings[key1]
|
| 105 |
+
f2 = merged_findings[key2]
|
| 106 |
+
|
| 107 |
+
if self._is_category_duplicate(f1, f2):
|
| 108 |
+
self.dedup_stats["category_merges"] += 1
|
| 109 |
+
self._merge_into(f1, f2)
|
| 110 |
+
to_remove.add(key2)
|
| 111 |
+
|
| 112 |
+
# Build final result
|
| 113 |
+
for key in keys_list:
|
| 114 |
+
if key not in to_remove:
|
| 115 |
+
result.append(merged_findings[key])
|
| 116 |
+
|
| 117 |
+
self.dedup_stats["final_count"] = len(result)
|
| 118 |
+
logger.info(
|
| 119 |
+
f"Deduplication: {len(findings)} → {len(result)} findings "
|
| 120 |
+
f"(removed {len(findings) - len(result)} duplicates)"
|
| 121 |
+
)
|
| 122 |
+
return result
|
| 123 |
+
|
| 124 |
+
def _exact_key(self, finding: Dict[str, Any]) -> str:
|
| 125 |
+
"""Generate an exact deduplication key.
|
| 126 |
+
|
| 127 |
+
Args:
|
| 128 |
+
finding: Finding dictionary.
|
| 129 |
+
|
| 130 |
+
Returns:
|
| 131 |
+
String key for exact matching.
|
| 132 |
+
"""
|
| 133 |
+
file_path = finding.get("file", "")
|
| 134 |
+
line = finding.get("line", 0)
|
| 135 |
+
category = finding.get("category", "").lower()
|
| 136 |
+
title = finding.get("title", "").lower().strip()
|
| 137 |
+
|
| 138 |
+
# Normalize the title
|
| 139 |
+
title = re.sub(r'[^a-z0-9]', '', title)
|
| 140 |
+
|
| 141 |
+
return f"{file_path}:{line}:{category}:{title}"
|
| 142 |
+
|
| 143 |
+
def _is_fuzzy_duplicate(self, f1: Dict[str, Any], f2: Dict[str, Any]) -> bool:
|
| 144 |
+
"""Check if two findings are fuzzy duplicates.
|
| 145 |
+
|
| 146 |
+
Two findings are fuzzy duplicates if:
|
| 147 |
+
- They are in the same file
|
| 148 |
+
- Their titles are similar (above threshold)
|
| 149 |
+
- Their line numbers are within 5 lines of each other
|
| 150 |
+
|
| 151 |
+
Args:
|
| 152 |
+
f1: First finding.
|
| 153 |
+
f2: Second finding.
|
| 154 |
+
|
| 155 |
+
Returns:
|
| 156 |
+
True if they are likely duplicates.
|
| 157 |
+
"""
|
| 158 |
+
# Must be in the same file
|
| 159 |
+
if f1.get("file", "") != f2.get("file", ""):
|
| 160 |
+
return False
|
| 161 |
+
|
| 162 |
+
# Check line proximity (within 5 lines)
|
| 163 |
+
line1 = f1.get("line", 0)
|
| 164 |
+
line2 = f2.get("line", 0)
|
| 165 |
+
if isinstance(line1, int) and isinstance(line2, int):
|
| 166 |
+
if abs(line1 - line2) > 5:
|
| 167 |
+
return False
|
| 168 |
+
|
| 169 |
+
# Check title similarity
|
| 170 |
+
title1 = f1.get("title", "").lower()
|
| 171 |
+
title2 = f2.get("title", "").lower()
|
| 172 |
+
similarity = SequenceMatcher(None, title1, title2).ratio()
|
| 173 |
+
|
| 174 |
+
if similarity >= self.similarity_threshold:
|
| 175 |
+
return True
|
| 176 |
+
|
| 177 |
+
# Also check category match with high title similarity
|
| 178 |
+
if (f1.get("category", "").lower() == f2.get("category", "").lower() and
|
| 179 |
+
similarity >= self.similarity_threshold * 0.8):
|
| 180 |
+
return True
|
| 181 |
+
|
| 182 |
+
return False
|
| 183 |
+
|
| 184 |
+
def _is_category_duplicate(self, f1: Dict[str, Any], f2: Dict[str, Any]) -> bool:
|
| 185 |
+
"""Check if two findings are category-level duplicates.
|
| 186 |
+
|
| 187 |
+
Same vulnerability type at nearby lines in the same file.
|
| 188 |
+
|
| 189 |
+
Args:
|
| 190 |
+
f1: First finding.
|
| 191 |
+
f2: Second finding.
|
| 192 |
+
|
| 193 |
+
Returns:
|
| 194 |
+
True if they are category-level duplicates.
|
| 195 |
+
"""
|
| 196 |
+
# Must be in same file and same category
|
| 197 |
+
if (f1.get("file", "") != f2.get("file", "") or
|
| 198 |
+
f1.get("category", "").lower() != f2.get("category", "").lower()):
|
| 199 |
+
return False
|
| 200 |
+
|
| 201 |
+
# Must be within 3 lines
|
| 202 |
+
line1 = f1.get("line", 0)
|
| 203 |
+
line2 = f2.get("line", 0)
|
| 204 |
+
if isinstance(line1, int) and isinstance(line2, int):
|
| 205 |
+
if abs(line1 - line2) <= 3:
|
| 206 |
+
return True
|
| 207 |
+
|
| 208 |
+
return False
|
| 209 |
+
|
| 210 |
+
def _merge_into(self, target: Dict[str, Any], source: Dict[str, Any]) -> None:
|
| 211 |
+
"""Merge source finding into target finding.
|
| 212 |
+
|
| 213 |
+
Keeps the higher severity, combines sources, and adds alternate references.
|
| 214 |
+
|
| 215 |
+
Args:
|
| 216 |
+
target: Target finding (modified in place).
|
| 217 |
+
source: Source finding to merge from.
|
| 218 |
+
"""
|
| 219 |
+
# Merge sources
|
| 220 |
+
if "sources" not in target:
|
| 221 |
+
target["sources"] = [target.get("source", target.get("agent", "unknown"))]
|
| 222 |
+
|
| 223 |
+
source_name = source.get("source", source.get("agent", "unknown"))
|
| 224 |
+
if source_name not in target["sources"]:
|
| 225 |
+
target["sources"].append(source_name)
|
| 226 |
+
|
| 227 |
+
# Keep higher severity
|
| 228 |
+
severity_order = {"CRITICAL": 4, "HIGH": 3, "MEDIUM": 2, "LOW": 1, "INFO": 0}
|
| 229 |
+
target_sev = severity_order.get(target.get("severity", "MEDIUM").upper(), 2)
|
| 230 |
+
source_sev = severity_order.get(source.get("severity", "MEDIUM").upper(), 2)
|
| 231 |
+
|
| 232 |
+
if source_sev > target_sev:
|
| 233 |
+
target["severity"] = source.get("severity", "MEDIUM")
|
| 234 |
+
|
| 235 |
+
# Keep higher confidence
|
| 236 |
+
target_conf = target.get("confidence", 0.8)
|
| 237 |
+
source_conf = source.get("confidence", 0.8)
|
| 238 |
+
target["confidence"] = max(target_conf, source_conf)
|
| 239 |
+
|
| 240 |
+
# Merge descriptions if different
|
| 241 |
+
if source.get("description", "") != target.get("description", ""):
|
| 242 |
+
target.setdefault("alternate_descriptions", [])
|
| 243 |
+
target["alternate_descriptions"].append(source.get("description", ""))
|
| 244 |
+
|
| 245 |
+
# Mark as merged
|
| 246 |
+
target["deduplicated"] = True
|
| 247 |
+
|
| 248 |
+
def get_stats(self) -> Dict[str, int]:
|
| 249 |
+
"""Get deduplication statistics.
|
| 250 |
+
|
| 251 |
+
Returns:
|
| 252 |
+
Dictionary of deduplication statistics.
|
| 253 |
+
"""
|
| 254 |
+
return self.dedup_stats.copy()
|
shield_agents/knowledge/__init__.py
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
"""Security knowledge bases for Shield Agents."""
|
shield_agents/knowledge/cve.py
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""CVE database interface for Shield Agents."""
|
| 2 |
+
|
| 3 |
+
# Known CVEs relevant to common vulnerabilities
|
| 4 |
+
KNOWN_CVES = {
|
| 5 |
+
"CVE-2021-44228": {
|
| 6 |
+
"title": "Log4Shell - Remote Code Injection in Log4j",
|
| 7 |
+
"severity": "CRITICAL",
|
| 8 |
+
"affected": "Apache Log4j 2.0-beta9 through 2.14.1",
|
| 9 |
+
"description": "Remote code execution via JNDI lookups in Log4j",
|
| 10 |
+
},
|
| 11 |
+
"CVE-2022-22965": {
|
| 12 |
+
"title": "Spring4Shell - RCE in Spring Framework",
|
| 13 |
+
"severity": "CRITICAL",
|
| 14 |
+
"affected": "Spring Framework 5.3.0 to 5.3.17, 5.2.0 to 5.2.19",
|
| 15 |
+
"description": "Remote code execution via data binding",
|
| 16 |
+
},
|
| 17 |
+
"CVE-2023-44487": {
|
| 18 |
+
"title": "HTTP/2 Rapid Reset Attack",
|
| 19 |
+
"severity": "HIGH",
|
| 20 |
+
"affected": "Multiple HTTP/2 implementations",
|
| 21 |
+
"description": "Denial of service via HTTP/2 stream cancellation",
|
| 22 |
+
},
|
| 23 |
+
"CVE-2021-41773": {
|
| 24 |
+
"title": "Apache HTTP Server Path Traversal",
|
| 25 |
+
"severity": "HIGH",
|
| 26 |
+
"affected": "Apache HTTP Server 2.4.49",
|
| 27 |
+
"description": "Path traversal and file disclosure",
|
| 28 |
+
},
|
| 29 |
+
"CVE-2020-9484": {
|
| 30 |
+
"title": "Apache Tomcat Session Deserialization",
|
| 31 |
+
"severity": "HIGH",
|
| 32 |
+
"affected": "Apache Tomcat 10.0.0-M1 to 10.0.0-M4, 9.0.0.M1 to 9.0.34",
|
| 33 |
+
"description": "Deserialization of session data leading to RCE",
|
| 34 |
+
},
|
| 35 |
+
}
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def lookup_cve(cve_id: str) -> dict:
|
| 39 |
+
"""Look up a CVE by ID.
|
| 40 |
+
|
| 41 |
+
Args:
|
| 42 |
+
cve_id: CVE identifier (e.g., 'CVE-2021-44228').
|
| 43 |
+
|
| 44 |
+
Returns:
|
| 45 |
+
CVE information dictionary, or empty dict if not found.
|
| 46 |
+
"""
|
| 47 |
+
return KNOWN_CVES.get(cve_id, {})
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def search_cves(keyword: str) -> list:
|
| 51 |
+
"""Search for CVEs matching a keyword.
|
| 52 |
+
|
| 53 |
+
Args:
|
| 54 |
+
keyword: Search keyword.
|
| 55 |
+
|
| 56 |
+
Returns:
|
| 57 |
+
List of matching CVE dictionaries.
|
| 58 |
+
"""
|
| 59 |
+
results = []
|
| 60 |
+
keyword_lower = keyword.lower()
|
| 61 |
+
for cve_id, info in KNOWN_CVES.items():
|
| 62 |
+
if (keyword_lower in info["title"].lower() or
|
| 63 |
+
keyword_lower in info["description"].lower()):
|
| 64 |
+
results.append({"id": cve_id, **info})
|
| 65 |
+
return results
|
shield_agents/knowledge/mitre.py
ADDED
|
@@ -0,0 +1,61 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""MITRE ATT&CK knowledge base for Shield Agents."""
|
| 2 |
+
|
| 3 |
+
MITRE_TECHNIQUES = {
|
| 4 |
+
"T1190": {
|
| 5 |
+
"name": "Exploit Public-Facing Application",
|
| 6 |
+
"tactic": "Initial Access",
|
| 7 |
+
"description": "Adversaries may exploit weaknesses in public-facing applications.",
|
| 8 |
+
},
|
| 9 |
+
"T1059": {
|
| 10 |
+
"name": "Command and Scripting Interpreter",
|
| 11 |
+
"tactic": "Execution",
|
| 12 |
+
"description": "Adversaries may abuse command and script interpreters.",
|
| 13 |
+
},
|
| 14 |
+
"T1078": {
|
| 15 |
+
"name": "Valid Accounts",
|
| 16 |
+
"tactic": "Defense Evasion",
|
| 17 |
+
"description": "Adversaries may use stolen credentials.",
|
| 18 |
+
},
|
| 19 |
+
"T1110": {
|
| 20 |
+
"name": "Brute Force",
|
| 21 |
+
"tactic": "Credential Access",
|
| 22 |
+
"description": "Adversaries may use brute force techniques.",
|
| 23 |
+
},
|
| 24 |
+
"T1053": {
|
| 25 |
+
"name": "Scheduled Task/Job",
|
| 26 |
+
"tactic": "Execution",
|
| 27 |
+
"description": "Adversaries may abuse task scheduling.",
|
| 28 |
+
},
|
| 29 |
+
"T1041": {
|
| 30 |
+
"name": "Exfiltration Over C2 Channel",
|
| 31 |
+
"tactic": "Exfiltration",
|
| 32 |
+
"description": "Adversaries may steal data through C2 channel.",
|
| 33 |
+
},
|
| 34 |
+
"T1566": {
|
| 35 |
+
"name": "Phishing",
|
| 36 |
+
"tactic": "Initial Access",
|
| 37 |
+
"description": "Adversaries may use phishing to gain access.",
|
| 38 |
+
},
|
| 39 |
+
"T1071": {
|
| 40 |
+
"name": "Application Layer Protocol",
|
| 41 |
+
"tactic": "Command and Control",
|
| 42 |
+
"description": "Adversaries may use application layer protocols for C2.",
|
| 43 |
+
},
|
| 44 |
+
}
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def get_mitre_technique(vulnerability_type: str) -> dict:
|
| 48 |
+
"""Get MITRE ATT&CK technique for a vulnerability type."""
|
| 49 |
+
mapping = {
|
| 50 |
+
"sql_injection": "T1190",
|
| 51 |
+
"xss": "T1190",
|
| 52 |
+
"command_injection": "T1059",
|
| 53 |
+
"credential_theft": "T1110",
|
| 54 |
+
"brute_force": "T1110",
|
| 55 |
+
"phishing": "T1566",
|
| 56 |
+
"exfiltration": "T1041",
|
| 57 |
+
"c2_communication": "T1071",
|
| 58 |
+
"injection": "T1190",
|
| 59 |
+
}
|
| 60 |
+
tech_id = mapping.get(vulnerability_type.lower(), "T1190")
|
| 61 |
+
return MITRE_TECHNIQUES.get(tech_id, MITRE_TECHNIQUES["T1190"])
|
shield_agents/knowledge/owasp.py
ADDED
|
@@ -0,0 +1,85 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""OWASP Top 10 knowledge base for Shield Agents."""
|
| 2 |
+
|
| 3 |
+
OWASP_TOP_10_2021 = {
|
| 4 |
+
"A01": {
|
| 5 |
+
"name": "Broken Access Control",
|
| 6 |
+
"description": "Access control enforces policy such that users cannot act outside their intended permissions.",
|
| 7 |
+
"url": "https://owasp.org/Top10/A01_2021-Broken_Access_Control/",
|
| 8 |
+
},
|
| 9 |
+
"A02": {
|
| 10 |
+
"name": "Cryptographic Failures",
|
| 11 |
+
"description": "Failures related to cryptography which often lead to sensitive data exposure.",
|
| 12 |
+
"url": "https://owasp.org/Top10/A02_2021-Cryptographic_Failures/",
|
| 13 |
+
},
|
| 14 |
+
"A03": {
|
| 15 |
+
"name": "Injection",
|
| 16 |
+
"description": "SQL injection, NoSQL injection, OS command injection, and LDAP injection.",
|
| 17 |
+
"url": "https://owasp.org/Top10/A03_2021-Injection/",
|
| 18 |
+
},
|
| 19 |
+
"A04": {
|
| 20 |
+
"name": "Insecure Design",
|
| 21 |
+
"description": "Missing or ineffective security controls and architecture flaws.",
|
| 22 |
+
"url": "https://owasp.org/Top10/A04_2021-Insecure_Design/",
|
| 23 |
+
},
|
| 24 |
+
"A05": {
|
| 25 |
+
"name": "Security Misconfiguration",
|
| 26 |
+
"description": "Missing security hardening, unnecessary features enabled, default accounts.",
|
| 27 |
+
"url": "https://owasp.org/Top10/A05_2021-Security_Misconfiguration/",
|
| 28 |
+
},
|
| 29 |
+
"A06": {
|
| 30 |
+
"name": "Vulnerable and Outdated Components",
|
| 31 |
+
"description": "Using components with known vulnerabilities.",
|
| 32 |
+
"url": "https://owasp.org/Top10/A06_2021-Vulnerable_and_Outdated_Components/",
|
| 33 |
+
},
|
| 34 |
+
"A07": {
|
| 35 |
+
"name": "Identification and Authentication Failures",
|
| 36 |
+
"description": "Authentication and session management weaknesses.",
|
| 37 |
+
"url": "https://owasp.org/Top10/A07_2021-Identification_and_Authentication_Failures/",
|
| 38 |
+
},
|
| 39 |
+
"A08": {
|
| 40 |
+
"name": "Software and Data Integrity Failures",
|
| 41 |
+
"description": "Code and infrastructure not protected against integrity violations.",
|
| 42 |
+
"url": "https://owasp.org/Top10/A08_2021-Software_and_Data_Integrity_Failures/",
|
| 43 |
+
},
|
| 44 |
+
"A09": {
|
| 45 |
+
"name": "Security Logging and Monitoring Failures",
|
| 46 |
+
"description": "Insufficient logging, monitoring, and alerting.",
|
| 47 |
+
"url": "https://owasp.org/Top10/A09_2021-Security_Logging_and_Monitoring_Failures/",
|
| 48 |
+
},
|
| 49 |
+
"A10": {
|
| 50 |
+
"name": "Server-Side Request Forgery",
|
| 51 |
+
"description": "Application fetches remote resources without validating user-supplied URLs.",
|
| 52 |
+
"url": "https://owasp.org/Top10/A10_2021-Server-Side_Request_Forgery_%28SSRF%29/",
|
| 53 |
+
},
|
| 54 |
+
}
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def get_owasp_category(vulnerability_type: str) -> dict:
|
| 58 |
+
"""Get OWASP category for a vulnerability type.
|
| 59 |
+
|
| 60 |
+
Args:
|
| 61 |
+
vulnerability_type: Type of vulnerability (e.g., 'sql_injection', 'xss').
|
| 62 |
+
|
| 63 |
+
Returns:
|
| 64 |
+
OWASP category dictionary.
|
| 65 |
+
"""
|
| 66 |
+
mapping = {
|
| 67 |
+
"sql_injection": "A03",
|
| 68 |
+
"xss": "A03",
|
| 69 |
+
"command_injection": "A03",
|
| 70 |
+
"path_traversal": "A01",
|
| 71 |
+
"broken_access_control": "A01",
|
| 72 |
+
"sensitive_data_exposure": "A02",
|
| 73 |
+
"crypto_failure": "A02",
|
| 74 |
+
"security_misconfiguration": "A05",
|
| 75 |
+
"auth_failure": "A07",
|
| 76 |
+
"ssrf": "A10",
|
| 77 |
+
"deserialization": "A08",
|
| 78 |
+
"outdated_component": "A06",
|
| 79 |
+
"logging_failure": "A09",
|
| 80 |
+
"injection": "A03",
|
| 81 |
+
"insecure_design": "A04",
|
| 82 |
+
}
|
| 83 |
+
|
| 84 |
+
category_id = mapping.get(vulnerability_type.lower(), "A03")
|
| 85 |
+
return OWASP_TOP_10_2021.get(category_id, OWASP_TOP_10_2021["A03"])
|
shield_agents/llm.py
ADDED
|
@@ -0,0 +1,794 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
LLM Provider abstraction for Shield Agents.
|
| 3 |
+
|
| 4 |
+
Supports multiple providers: OpenAI, Anthropic, Ollama, and a smart Mock provider.
|
| 5 |
+
Includes robust fallback parsing for invalid LLM JSON responses (~30% failure rate).
|
| 6 |
+
"""
|
| 7 |
+
|
| 8 |
+
import json
|
| 9 |
+
import re
|
| 10 |
+
import logging
|
| 11 |
+
from abc import ABC, abstractmethod
|
| 12 |
+
from typing import Any, Dict, List, Optional
|
| 13 |
+
|
| 14 |
+
from .config import LLMConfig
|
| 15 |
+
|
| 16 |
+
logger = logging.getLogger("shield_agents.llm")
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
# =============================================================================
|
| 20 |
+
# FEATURE: Real LLM Fallback Handling
|
| 21 |
+
# Robust parsing for when LLMs return invalid JSON (~30% of the time)
|
| 22 |
+
# =============================================================================
|
| 23 |
+
|
| 24 |
+
class LLMResponseParser:
|
| 25 |
+
"""Parse LLM responses with multiple fallback strategies.
|
| 26 |
+
|
| 27 |
+
Handles these common failure modes:
|
| 28 |
+
1. Valid JSON - direct parse
|
| 29 |
+
2. JSON wrapped in markdown code blocks (```json ... ```)
|
| 30 |
+
3. JSON with trailing commas
|
| 31 |
+
4. JSON with single quotes instead of double quotes
|
| 32 |
+
5. JSON embedded in explanatory text
|
| 33 |
+
6. Partial JSON that can be repaired
|
| 34 |
+
7. Non-JSON structured text (bullet lists, etc.)
|
| 35 |
+
"""
|
| 36 |
+
|
| 37 |
+
@staticmethod
|
| 38 |
+
def parse(response_text: str) -> Dict[str, Any]:
|
| 39 |
+
"""Parse an LLM response with progressive fallback strategies.
|
| 40 |
+
|
| 41 |
+
Args:
|
| 42 |
+
response_text: Raw text response from the LLM.
|
| 43 |
+
|
| 44 |
+
Returns:
|
| 45 |
+
Parsed dictionary, or empty dict if all strategies fail.
|
| 46 |
+
"""
|
| 47 |
+
if not response_text or not response_text.strip():
|
| 48 |
+
return {}
|
| 49 |
+
|
| 50 |
+
# Strategy 1: Direct JSON parse
|
| 51 |
+
result = LLMResponseParser._try_direct_json(response_text)
|
| 52 |
+
if result is not None:
|
| 53 |
+
return result
|
| 54 |
+
|
| 55 |
+
# Strategy 2: Extract from markdown code blocks
|
| 56 |
+
result = LLMResponseParser._try_code_block_extraction(response_text)
|
| 57 |
+
if result is not None:
|
| 58 |
+
return result
|
| 59 |
+
|
| 60 |
+
# Strategy 3: Fix common JSON issues
|
| 61 |
+
result = LLMResponseParser._try_json_repair(response_text)
|
| 62 |
+
if result is not None:
|
| 63 |
+
return result
|
| 64 |
+
|
| 65 |
+
# Strategy 4: Find JSON-like structure in text
|
| 66 |
+
result = LLMResponseParser._try_embedded_json(response_text)
|
| 67 |
+
if result is not None:
|
| 68 |
+
return result
|
| 69 |
+
|
| 70 |
+
# Strategy 5: Parse structured text into findings
|
| 71 |
+
result = LLMResponseParser._try_text_to_structured(response_text)
|
| 72 |
+
if result is not None:
|
| 73 |
+
return result
|
| 74 |
+
|
| 75 |
+
logger.warning("All LLM response parsing strategies failed")
|
| 76 |
+
return {}
|
| 77 |
+
|
| 78 |
+
@staticmethod
|
| 79 |
+
def _try_direct_json(text: str) -> Optional[Dict]:
|
| 80 |
+
"""Strategy 1: Try direct JSON parsing."""
|
| 81 |
+
try:
|
| 82 |
+
result = json.loads(text.strip())
|
| 83 |
+
if isinstance(result, dict):
|
| 84 |
+
return result
|
| 85 |
+
if isinstance(result, list) and len(result) > 0:
|
| 86 |
+
return {"findings": result}
|
| 87 |
+
except json.JSONDecodeError:
|
| 88 |
+
pass
|
| 89 |
+
return None
|
| 90 |
+
|
| 91 |
+
@staticmethod
|
| 92 |
+
def _try_code_block_extraction(text: str) -> Optional[Dict]:
|
| 93 |
+
"""Strategy 2: Extract JSON from markdown code blocks."""
|
| 94 |
+
patterns = [
|
| 95 |
+
r"```json\s*\n(.*?)\n\s*```",
|
| 96 |
+
r"```\s*\n(.*?)\n\s*```",
|
| 97 |
+
r"`(.*?)`",
|
| 98 |
+
]
|
| 99 |
+
for pattern in patterns:
|
| 100 |
+
matches = re.findall(pattern, text, re.DOTALL)
|
| 101 |
+
for match in matches:
|
| 102 |
+
try:
|
| 103 |
+
result = json.loads(match.strip())
|
| 104 |
+
if isinstance(result, dict):
|
| 105 |
+
return result
|
| 106 |
+
if isinstance(result, list):
|
| 107 |
+
return {"findings": result}
|
| 108 |
+
except json.JSONDecodeError:
|
| 109 |
+
continue
|
| 110 |
+
return None
|
| 111 |
+
|
| 112 |
+
@staticmethod
|
| 113 |
+
def _try_json_repair(text: str) -> Optional[Dict]:
|
| 114 |
+
"""Strategy 3: Fix common JSON issues and retry."""
|
| 115 |
+
# Extract what looks like JSON content
|
| 116 |
+
json_candidates = []
|
| 117 |
+
|
| 118 |
+
# Try the whole text
|
| 119 |
+
json_candidates.append(text.strip())
|
| 120 |
+
|
| 121 |
+
# Try content between first { and last }
|
| 122 |
+
brace_match = re.search(r'\{.*\}', text, re.DOTALL)
|
| 123 |
+
if brace_match:
|
| 124 |
+
json_candidates.append(brace_match.group(0))
|
| 125 |
+
|
| 126 |
+
# Try content between first [ and last ]
|
| 127 |
+
bracket_match = re.search(r'\[.*\]', text, re.DOTALL)
|
| 128 |
+
if bracket_match:
|
| 129 |
+
json_candidates.append(bracket_match.group(0))
|
| 130 |
+
|
| 131 |
+
for candidate in json_candidates:
|
| 132 |
+
repaired = candidate
|
| 133 |
+
# Fix trailing commas before } or ]
|
| 134 |
+
repaired = re.sub(r',\s*([}\]])', r'\1', repaired)
|
| 135 |
+
# Fix single quotes to double quotes (careful with apostrophes)
|
| 136 |
+
# Only replace single quotes that look like JSON delimiters
|
| 137 |
+
repaired = re.sub(r"(?<![a-zA-Z])'([^']*?)'(?![a-zA-Z])", r'"\1"', repaired)
|
| 138 |
+
# Fix unquoted keys
|
| 139 |
+
repaired = re.sub(r'(\w+)\s*:', r'"\1":', repaired)
|
| 140 |
+
# Fix double-quoted keys that got re-quoted
|
| 141 |
+
repaired = re.sub(r'""(\w+)"":', r'"\1":', repaired)
|
| 142 |
+
# Remove comments
|
| 143 |
+
repaired = re.sub(r'//.*?\n', '\n', repaired)
|
| 144 |
+
repaired = re.sub(r'/\*.*?\*/', '', repaired, flags=re.DOTALL)
|
| 145 |
+
|
| 146 |
+
try:
|
| 147 |
+
result = json.loads(repaired)
|
| 148 |
+
if isinstance(result, dict):
|
| 149 |
+
return result
|
| 150 |
+
if isinstance(result, list):
|
| 151 |
+
return {"findings": result}
|
| 152 |
+
except json.JSONDecodeError:
|
| 153 |
+
continue
|
| 154 |
+
|
| 155 |
+
return None
|
| 156 |
+
|
| 157 |
+
@staticmethod
|
| 158 |
+
def _try_embedded_json(text: str) -> Optional[Dict]:
|
| 159 |
+
"""Strategy 4: Find JSON-like structure embedded in text."""
|
| 160 |
+
# Look for patterns like "findings": [...] or "vulnerabilities": [...]
|
| 161 |
+
patterns = [
|
| 162 |
+
r'"findings"\s*:\s*(\[.*?\])',
|
| 163 |
+
r'"vulnerabilities"\s*:\s*(\[.*?\])',
|
| 164 |
+
r'"results"\s*:\s*(\[.*?\])',
|
| 165 |
+
r'"issues"\s*:\s*(\[.*?\])',
|
| 166 |
+
]
|
| 167 |
+
for pattern in patterns:
|
| 168 |
+
match = re.search(pattern, text, re.DOTALL)
|
| 169 |
+
if match:
|
| 170 |
+
try:
|
| 171 |
+
findings = json.loads(match.group(1))
|
| 172 |
+
return {"findings": findings}
|
| 173 |
+
except json.JSONDecodeError:
|
| 174 |
+
continue
|
| 175 |
+
return None
|
| 176 |
+
|
| 177 |
+
@staticmethod
|
| 178 |
+
def _try_text_to_structured(text: str) -> Optional[Dict]:
|
| 179 |
+
"""Strategy 5: Parse structured text (bullet lists, numbered items) into findings."""
|
| 180 |
+
findings = []
|
| 181 |
+
|
| 182 |
+
# Match patterns like:
|
| 183 |
+
# - **Vulnerability**: SQL Injection in login.py:42
|
| 184 |
+
# - [HIGH] XSS vulnerability found in search.py
|
| 185 |
+
# 1. SQL Injection - user input not sanitized
|
| 186 |
+
bullet_patterns = [
|
| 187 |
+
r'(?:^|\n)\s*[-*]\s*\**\s*(.+?)(?:\n|$)',
|
| 188 |
+
r'(?:^|\n)\s*\d+\.\s*(.+?)(?:\n|$)',
|
| 189 |
+
r'(?:^|\n)\s*\[(HIGH|MEDIUM|LOW|CRITICAL)\]\s*(.+?)(?:\n|$)',
|
| 190 |
+
]
|
| 191 |
+
|
| 192 |
+
for pattern in bullet_patterns:
|
| 193 |
+
matches = re.findall(pattern, text, re.MULTILINE)
|
| 194 |
+
for match in matches:
|
| 195 |
+
if isinstance(match, tuple):
|
| 196 |
+
severity = match[0] if match[0] in ("HIGH", "MEDIUM", "LOW", "CRITICAL") else "MEDIUM"
|
| 197 |
+
description = match[1] if len(match) > 1 else match[0]
|
| 198 |
+
else:
|
| 199 |
+
severity = "MEDIUM"
|
| 200 |
+
description = match
|
| 201 |
+
|
| 202 |
+
if len(description.strip()) > 5:
|
| 203 |
+
findings.append({
|
| 204 |
+
"title": description.strip()[:100],
|
| 205 |
+
"description": description.strip(),
|
| 206 |
+
"severity": severity,
|
| 207 |
+
"source": "llm_text_parse",
|
| 208 |
+
})
|
| 209 |
+
|
| 210 |
+
if findings:
|
| 211 |
+
return {"findings": findings}
|
| 212 |
+
return None
|
| 213 |
+
|
| 214 |
+
|
| 215 |
+
# =============================================================================
|
| 216 |
+
# LLM Provider Base and Implementations
|
| 217 |
+
# =============================================================================
|
| 218 |
+
|
| 219 |
+
class BaseLLMProvider(ABC):
|
| 220 |
+
"""Abstract base class for LLM providers."""
|
| 221 |
+
|
| 222 |
+
def __init__(self, config: LLMConfig):
|
| 223 |
+
self.config = config
|
| 224 |
+
|
| 225 |
+
@abstractmethod
|
| 226 |
+
async def complete(self, messages: List[Dict[str, str]], **kwargs) -> str:
|
| 227 |
+
"""Send messages and return raw text response."""
|
| 228 |
+
pass
|
| 229 |
+
|
| 230 |
+
@abstractmethod
|
| 231 |
+
async def complete_json(self, messages: List[Dict[str, str]], **kwargs) -> Dict[str, Any]:
|
| 232 |
+
"""Send messages and return parsed JSON response with fallback handling."""
|
| 233 |
+
pass
|
| 234 |
+
|
| 235 |
+
|
| 236 |
+
class OpenAIProvider(BaseLLMProvider):
|
| 237 |
+
"""OpenAI API provider."""
|
| 238 |
+
|
| 239 |
+
def __init__(self, config: LLMConfig):
|
| 240 |
+
super().__init__(config)
|
| 241 |
+
|
| 242 |
+
async def complete(self, messages: List[Dict[str, str]], **kwargs) -> str:
|
| 243 |
+
try:
|
| 244 |
+
import openai
|
| 245 |
+
client = openai.AsyncOpenAI(
|
| 246 |
+
api_key=self.config.api_key,
|
| 247 |
+
base_url=self.config.base_url,
|
| 248 |
+
)
|
| 249 |
+
response = await client.chat.completions.create(
|
| 250 |
+
model=self.config.model,
|
| 251 |
+
messages=messages,
|
| 252 |
+
temperature=self.config.temperature,
|
| 253 |
+
max_tokens=self.config.max_tokens,
|
| 254 |
+
timeout=self.config.timeout,
|
| 255 |
+
**kwargs,
|
| 256 |
+
)
|
| 257 |
+
return response.choices[0].message.content or ""
|
| 258 |
+
except ImportError:
|
| 259 |
+
logger.error("openai package not installed. Run: pip install openai")
|
| 260 |
+
return ""
|
| 261 |
+
except Exception as e:
|
| 262 |
+
logger.error(f"OpenAI API error: {e}")
|
| 263 |
+
return ""
|
| 264 |
+
|
| 265 |
+
async def complete_json(self, messages: List[Dict[str, str]], **kwargs) -> Dict[str, Any]:
|
| 266 |
+
raw = await self.complete(messages, **kwargs)
|
| 267 |
+
if not raw:
|
| 268 |
+
return {}
|
| 269 |
+
return LLMResponseParser.parse(raw)
|
| 270 |
+
|
| 271 |
+
|
| 272 |
+
class AnthropicProvider(BaseLLMProvider):
|
| 273 |
+
"""Anthropic API provider."""
|
| 274 |
+
|
| 275 |
+
def __init__(self, config: LLMConfig):
|
| 276 |
+
super().__init__(config)
|
| 277 |
+
|
| 278 |
+
async def complete(self, messages: List[Dict[str, str]], **kwargs) -> str:
|
| 279 |
+
try:
|
| 280 |
+
import anthropic
|
| 281 |
+
client = anthropic.AsyncAnthropic(api_key=self.config.api_key)
|
| 282 |
+
# Extract system message if present
|
| 283 |
+
system_msg = ""
|
| 284 |
+
user_messages = []
|
| 285 |
+
for msg in messages:
|
| 286 |
+
if msg["role"] == "system":
|
| 287 |
+
system_msg = msg["content"]
|
| 288 |
+
else:
|
| 289 |
+
user_messages.append(msg)
|
| 290 |
+
|
| 291 |
+
response = await client.messages.create(
|
| 292 |
+
model=self.config.model,
|
| 293 |
+
max_tokens=self.config.max_tokens,
|
| 294 |
+
system=system_msg,
|
| 295 |
+
messages=user_messages,
|
| 296 |
+
**kwargs,
|
| 297 |
+
)
|
| 298 |
+
return response.content[0].text if response.content else ""
|
| 299 |
+
except ImportError:
|
| 300 |
+
logger.error("anthropic package not installed. Run: pip install anthropic")
|
| 301 |
+
return ""
|
| 302 |
+
except Exception as e:
|
| 303 |
+
logger.error(f"Anthropic API error: {e}")
|
| 304 |
+
return ""
|
| 305 |
+
|
| 306 |
+
async def complete_json(self, messages: List[Dict[str, str]], **kwargs) -> Dict[str, Any]:
|
| 307 |
+
raw = await self.complete(messages, **kwargs)
|
| 308 |
+
if not raw:
|
| 309 |
+
return {}
|
| 310 |
+
return LLMResponseParser.parse(raw)
|
| 311 |
+
|
| 312 |
+
|
| 313 |
+
class OllamaProvider(BaseLLMProvider):
|
| 314 |
+
"""Ollama local LLM provider."""
|
| 315 |
+
|
| 316 |
+
def __init__(self, config: LLMConfig):
|
| 317 |
+
super().__init__(config)
|
| 318 |
+
if not self.config.base_url:
|
| 319 |
+
self.config.base_url = "http://localhost:11434"
|
| 320 |
+
|
| 321 |
+
async def complete(self, messages: List[Dict[str, str]], **kwargs) -> str:
|
| 322 |
+
try:
|
| 323 |
+
import httpx
|
| 324 |
+
async with httpx.AsyncClient(timeout=self.config.timeout) as client:
|
| 325 |
+
response = await client.post(
|
| 326 |
+
f"{self.config.base_url}/api/chat",
|
| 327 |
+
json={
|
| 328 |
+
"model": self.config.model,
|
| 329 |
+
"messages": messages,
|
| 330 |
+
"stream": False,
|
| 331 |
+
"options": {
|
| 332 |
+
"temperature": self.config.temperature,
|
| 333 |
+
"num_predict": self.config.max_tokens,
|
| 334 |
+
},
|
| 335 |
+
},
|
| 336 |
+
)
|
| 337 |
+
response.raise_for_status()
|
| 338 |
+
data = response.json()
|
| 339 |
+
return data.get("message", {}).get("content", "")
|
| 340 |
+
except ImportError:
|
| 341 |
+
logger.error("httpx package not installed. Run: pip install httpx")
|
| 342 |
+
return ""
|
| 343 |
+
except Exception as e:
|
| 344 |
+
logger.error(f"Ollama API error: {e}")
|
| 345 |
+
return ""
|
| 346 |
+
|
| 347 |
+
async def complete_json(self, messages: List[Dict[str, str]], **kwargs) -> Dict[str, Any]:
|
| 348 |
+
raw = await self.complete(messages, **kwargs)
|
| 349 |
+
if not raw:
|
| 350 |
+
return {}
|
| 351 |
+
return LLMResponseParser.parse(raw)
|
| 352 |
+
|
| 353 |
+
|
| 354 |
+
# =============================================================================
|
| 355 |
+
# FEATURE: Smarter Mock Provider
|
| 356 |
+
# Pattern-matched findings based on actual code content, not static JSON
|
| 357 |
+
# =============================================================================
|
| 358 |
+
|
| 359 |
+
class MockProvider(BaseLLMProvider):
|
| 360 |
+
"""Smart Mock LLM provider that returns pattern-matched findings.
|
| 361 |
+
|
| 362 |
+
Instead of returning static JSON, this provider analyzes the input code
|
| 363 |
+
and returns findings that actually match the scanned content. This makes
|
| 364 |
+
demo mode feel realistic and useful for testing.
|
| 365 |
+
"""
|
| 366 |
+
|
| 367 |
+
# Pattern definitions: (regex_pattern, finding_info)
|
| 368 |
+
VULNERABILITY_PATTERNS = [
|
| 369 |
+
(r"execute\s*\(|exec\s*\(", {
|
| 370 |
+
"title": "Dynamic Code Execution",
|
| 371 |
+
"severity": "CRITICAL",
|
| 372 |
+
"category": "injection",
|
| 373 |
+
"description": "Use of exec/execute can lead to code injection if user input is involved",
|
| 374 |
+
"remediation": "Avoid dynamic code execution. Use safe alternatives like ast.literal_eval() for parsing.",
|
| 375 |
+
"cwe": "CWE-94",
|
| 376 |
+
}),
|
| 377 |
+
(r"eval\s*\(", {
|
| 378 |
+
"title": "Unsafe eval() Usage",
|
| 379 |
+
"severity": "CRITICAL",
|
| 380 |
+
"category": "injection",
|
| 381 |
+
"description": "eval() can execute arbitrary code if user input is passed",
|
| 382 |
+
"remediation": "Replace eval() with ast.literal_eval() or specific parsing functions.",
|
| 383 |
+
"cwe": "CWE-95",
|
| 384 |
+
}),
|
| 385 |
+
(r"SELECT\s+.*\s+FROM\s+.*\s+WHERE.*\+\s*[\w]+|f['\"].*SELECT.*WHERE|format\s*\(.*SELECT", {
|
| 386 |
+
"title": "SQL Injection",
|
| 387 |
+
"severity": "CRITICAL",
|
| 388 |
+
"category": "injection",
|
| 389 |
+
"description": "SQL query constructed with string formatting or concatenation",
|
| 390 |
+
"remediation": "Use parameterized queries with placeholders instead of string formatting.",
|
| 391 |
+
"cwe": "CWE-89",
|
| 392 |
+
}),
|
| 393 |
+
(r"cursor\.execute\s*\(\s*[f'\"]|\.raw\s*\(\s*[f'\"]", {
|
| 394 |
+
"title": "SQL Injection via ORM Raw Query",
|
| 395 |
+
"severity": "CRITICAL",
|
| 396 |
+
"category": "injection",
|
| 397 |
+
"description": "Raw SQL query with potential string interpolation",
|
| 398 |
+
"remediation": "Use ORM query methods or parameterized raw queries.",
|
| 399 |
+
"cwe": "CWE-89",
|
| 400 |
+
}),
|
| 401 |
+
(r"innerHTML\s*=|document\.write\s*\(", {
|
| 402 |
+
"title": "Cross-Site Scripting (XSS)",
|
| 403 |
+
"severity": "HIGH",
|
| 404 |
+
"category": "xss",
|
| 405 |
+
"description": "Direct DOM manipulation can lead to XSS if content is not sanitized",
|
| 406 |
+
"remediation": "Use textContent instead of innerHTML, or sanitize with DOMPurify.",
|
| 407 |
+
"cwe": "CWE-79",
|
| 408 |
+
}),
|
| 409 |
+
(r"os\.system\s*\(|subprocess\.call\s*\(|subprocess\.Popen\s*\(", {
|
| 410 |
+
"title": "OS Command Injection",
|
| 411 |
+
"severity": "CRITICAL",
|
| 412 |
+
"category": "injection",
|
| 413 |
+
"description": "OS command execution can lead to command injection",
|
| 414 |
+
"remediation": "Use subprocess with shell=False and list arguments. Never pass user input to shell commands.",
|
| 415 |
+
"cwe": "CWE-78",
|
| 416 |
+
}),
|
| 417 |
+
(r"pickle\.loads?\s*\(", {
|
| 418 |
+
"title": "Unsafe Deserialization",
|
| 419 |
+
"severity": "CRITICAL",
|
| 420 |
+
"category": "deserialization",
|
| 421 |
+
"description": "pickle can execute arbitrary code during deserialization",
|
| 422 |
+
"remediation": "Use JSON or other safe serialization formats. If pickle is necessary, never deserialize untrusted data.",
|
| 423 |
+
"cwe": "CWE-502",
|
| 424 |
+
}),
|
| 425 |
+
(r"yaml\.load\s*\([^)]*\)(?!.*Loader)", {
|
| 426 |
+
"title": "Unsafe YAML Loading",
|
| 427 |
+
"severity": "HIGH",
|
| 428 |
+
"category": "deserialization",
|
| 429 |
+
"description": "yaml.load() without safe Loader can execute arbitrary code",
|
| 430 |
+
"remediation": "Use yaml.safe_load() or yaml.load(data, Loader=yaml.SafeLoader).",
|
| 431 |
+
"cwe": "CWE-502",
|
| 432 |
+
}),
|
| 433 |
+
(r"hashlib\.(md5|sha1)\s*\(", {
|
| 434 |
+
"title": "Weak Cryptographic Hash",
|
| 435 |
+
"severity": "MEDIUM",
|
| 436 |
+
"category": "cryptography",
|
| 437 |
+
"description": "MD5 and SHA-1 are cryptographically broken and should not be used for security purposes",
|
| 438 |
+
"remediation": "Use SHA-256 or stronger hashing algorithms.",
|
| 439 |
+
"cwe": "CWE-328",
|
| 440 |
+
}),
|
| 441 |
+
(r"assert\s+", {
|
| 442 |
+
"title": "Assertion Used in Security Context",
|
| 443 |
+
"severity": "LOW",
|
| 444 |
+
"category": "security-misconfiguration",
|
| 445 |
+
"description": "Assertions can be disabled with -O flag and should not be used for security checks",
|
| 446 |
+
"remediation": "Replace assertions with proper if/raise constructs for security validation.",
|
| 447 |
+
"cwe": "CWE-617",
|
| 448 |
+
}),
|
| 449 |
+
(r"random\.(random|randint|choice)\s*\(", {
|
| 450 |
+
"title": "Insecure Random Number Generator",
|
| 451 |
+
"severity": "MEDIUM",
|
| 452 |
+
"category": "cryptography",
|
| 453 |
+
"description": "random module is not cryptographically secure",
|
| 454 |
+
"remediation": "Use secrets module for security-sensitive random number generation.",
|
| 455 |
+
"cwe": "CWE-338",
|
| 456 |
+
}),
|
| 457 |
+
(r"ssl\._create_unverified_context|verify\s*=\s*False|CERT_NONE", {
|
| 458 |
+
"title": "SSL/TLS Verification Disabled",
|
| 459 |
+
"severity": "HIGH",
|
| 460 |
+
"category": "security-misconfiguration",
|
| 461 |
+
"description": "SSL certificate verification is disabled",
|
| 462 |
+
"remediation": "Always verify SSL certificates. Never use verify=False in production.",
|
| 463 |
+
"cwe": "CWE-295",
|
| 464 |
+
}),
|
| 465 |
+
(r"jwt\.decode\s*\([^)]*\)(?!.*algorithms)", {
|
| 466 |
+
"title": "JWT Algorithm Confusion",
|
| 467 |
+
"severity": "HIGH",
|
| 468 |
+
"category": "authentication",
|
| 469 |
+
"description": "JWT decode without specifying algorithms can lead to algorithm confusion attacks",
|
| 470 |
+
"remediation": "Always specify the expected algorithms parameter in jwt.decode().",
|
| 471 |
+
"cwe": "CWE-327",
|
| 472 |
+
}),
|
| 473 |
+
(r"cors\s*=\s*True|Access-Control-Allow-Origin.*\*", {
|
| 474 |
+
"title": "Overly Permissive CORS",
|
| 475 |
+
"severity": "MEDIUM",
|
| 476 |
+
"category": "security-misconfiguration",
|
| 477 |
+
"description": "CORS is configured to allow all origins",
|
| 478 |
+
"remediation": "Restrict CORS to specific trusted origins instead of using wildcard.",
|
| 479 |
+
"cwe": "CWE-942",
|
| 480 |
+
}),
|
| 481 |
+
(r"render_template_string\s*\(", {
|
| 482 |
+
"title": "Server-Side Template Injection (SSTI)",
|
| 483 |
+
"severity": "CRITICAL",
|
| 484 |
+
"category": "injection",
|
| 485 |
+
"description": "render_template_string with user input can lead to SSTI",
|
| 486 |
+
"remediation": "Never pass user input to template rendering. Use render_template with fixed templates.",
|
| 487 |
+
"cwe": "CWE-1336",
|
| 488 |
+
}),
|
| 489 |
+
(r"\.format\s*\(|f['\"]", {
|
| 490 |
+
"title": "Potential String Injection",
|
| 491 |
+
"severity": "MEDIUM",
|
| 492 |
+
"category": "injection",
|
| 493 |
+
"description": "String formatting with potential user-controlled input",
|
| 494 |
+
"remediation": "Validate and sanitize all user input before using in string formatting.",
|
| 495 |
+
"cwe": "CWE-134",
|
| 496 |
+
}),
|
| 497 |
+
]
|
| 498 |
+
|
| 499 |
+
# Agent-specific patterns for differentiated demo mode
|
| 500 |
+
# Each agent has its own perspective on the code
|
| 501 |
+
AGENT_PATTERNS = {
|
| 502 |
+
"VulnAgent": {
|
| 503 |
+
"focus": "vulnerability detection",
|
| 504 |
+
"extra_patterns": [
|
| 505 |
+
(r'cursor\.execute\s*\(|\.raw\s*\(', {
|
| 506 |
+
"title": "SQL Injection Risk",
|
| 507 |
+
"severity": "CRITICAL",
|
| 508 |
+
"category": "injection",
|
| 509 |
+
"description": "Database query execution with potential SQL injection",
|
| 510 |
+
"remediation": "Use parameterized queries with placeholders",
|
| 511 |
+
"cwe": "CWE-89",
|
| 512 |
+
}),
|
| 513 |
+
(r'eval\s*\(|exec\s*\(', {
|
| 514 |
+
"title": "Dynamic Code Execution",
|
| 515 |
+
"severity": "CRITICAL",
|
| 516 |
+
"category": "injection",
|
| 517 |
+
"description": "Dynamic code execution can lead to code injection",
|
| 518 |
+
"remediation": "Avoid eval/exec, use safe alternatives",
|
| 519 |
+
"cwe": "CWE-94",
|
| 520 |
+
}),
|
| 521 |
+
(r'os\.system\s*\(|subprocess\.\w+\s*\(.*shell\s*=\s*True', {
|
| 522 |
+
"title": "OS Command Injection",
|
| 523 |
+
"severity": "CRITICAL",
|
| 524 |
+
"category": "injection",
|
| 525 |
+
"description": "OS command execution with potential injection",
|
| 526 |
+
"remediation": "Use subprocess with shell=False and list arguments",
|
| 527 |
+
"cwe": "CWE-78",
|
| 528 |
+
}),
|
| 529 |
+
(r'pickle\.loads?\s*\(', {
|
| 530 |
+
"title": "Insecure Deserialization",
|
| 531 |
+
"severity": "CRITICAL",
|
| 532 |
+
"category": "deserialization",
|
| 533 |
+
"description": "Pickle deserialization can execute arbitrary code",
|
| 534 |
+
"remediation": "Use JSON serialization or yaml.safe_load()",
|
| 535 |
+
"cwe": "CWE-502",
|
| 536 |
+
}),
|
| 537 |
+
],
|
| 538 |
+
},
|
| 539 |
+
"ThreatAgent": {
|
| 540 |
+
"focus": "threat modeling and attack vectors",
|
| 541 |
+
"extra_patterns": [
|
| 542 |
+
(r'request\.(args|form|data|json)', {
|
| 543 |
+
"title": "User Input Entry Point",
|
| 544 |
+
"severity": "MEDIUM",
|
| 545 |
+
"category": "attack-surface",
|
| 546 |
+
"description": "HTTP request parameter accepted without validation - potential attack vector",
|
| 547 |
+
"remediation": "Validate and sanitize all user input at entry points",
|
| 548 |
+
"mitre_technique": "T1190",
|
| 549 |
+
}),
|
| 550 |
+
(r'@app\.route', {
|
| 551 |
+
"title": "Exposed HTTP Endpoint",
|
| 552 |
+
"severity": "LOW",
|
| 553 |
+
"category": "attack-surface",
|
| 554 |
+
"description": "HTTP endpoint exposed to network - part of attack surface",
|
| 555 |
+
"remediation": "Ensure endpoints have proper authentication and rate limiting",
|
| 556 |
+
"mitre_technique": "T1190",
|
| 557 |
+
}),
|
| 558 |
+
(r'debug\s*=\s*True', {
|
| 559 |
+
"title": "Debug Mode Enabled",
|
| 560 |
+
"severity": "HIGH",
|
| 561 |
+
"category": "security-misconfiguration",
|
| 562 |
+
"description": "Debug mode enabled in application - exposes sensitive information",
|
| 563 |
+
"remediation": "Disable debug mode in production (debug=False)",
|
| 564 |
+
"mitre_technique": "T1078",
|
| 565 |
+
}),
|
| 566 |
+
(r'import\s+os|import\s+subprocess', {
|
| 567 |
+
"title": "System Module Import",
|
| 568 |
+
"severity": "LOW",
|
| 569 |
+
"category": "attack-surface",
|
| 570 |
+
"description": "System-level module imported - increases attack surface if misused",
|
| 571 |
+
"remediation": "Ensure system modules are used safely and inputs are validated",
|
| 572 |
+
"mitre_technique": "T1059",
|
| 573 |
+
}),
|
| 574 |
+
],
|
| 575 |
+
},
|
| 576 |
+
"ReconAgent": {
|
| 577 |
+
"focus": "reconnaissance and information disclosure",
|
| 578 |
+
"extra_patterns": [
|
| 579 |
+
(r'print\s*\(|logger\.\w+\s*\(', {
|
| 580 |
+
"title": "Information Leakage via Logging",
|
| 581 |
+
"severity": "LOW",
|
| 582 |
+
"category": "information-disclosure",
|
| 583 |
+
"description": "Print/logging statements may expose sensitive data in production logs",
|
| 584 |
+
"remediation": "Remove debug prints, use appropriate log levels, never log secrets",
|
| 585 |
+
"cwe": "CWE-532",
|
| 586 |
+
}),
|
| 587 |
+
(r'#\s*TODO|#\s*FIXME|#\s*HACK|#\s*XXX', {
|
| 588 |
+
"title": "Developer Comment - Potential Security Debt",
|
| 589 |
+
"severity": "INFO",
|
| 590 |
+
"category": "information-disclosure",
|
| 591 |
+
"description": "Developer comments indicate unfinished security work or known issues",
|
| 592 |
+
"remediation": "Review and resolve all security-related TODO/FIXME comments",
|
| 593 |
+
}),
|
| 594 |
+
(r'debug\s*=\s*True', {
|
| 595 |
+
"title": "Debug Information Exposure",
|
| 596 |
+
"severity": "HIGH",
|
| 597 |
+
"category": "information-disclosure",
|
| 598 |
+
"description": "Debug mode exposes stack traces and environment variables",
|
| 599 |
+
"remediation": "Set debug=False in production",
|
| 600 |
+
"cwe": "CWE-200",
|
| 601 |
+
}),
|
| 602 |
+
(r'secret|password|token|key', {
|
| 603 |
+
"title": "Security-Sensitive Variable Names",
|
| 604 |
+
"severity": "MEDIUM",
|
| 605 |
+
"category": "information-disclosure",
|
| 606 |
+
"description": "Variables with security-sensitive names may indicate secret handling",
|
| 607 |
+
"remediation": "Ensure secrets are loaded from environment, not hardcoded",
|
| 608 |
+
"cwe": "CWE-798",
|
| 609 |
+
}),
|
| 610 |
+
],
|
| 611 |
+
},
|
| 612 |
+
"ComplianceAgent": {
|
| 613 |
+
"focus": "OWASP compliance and security standards",
|
| 614 |
+
"extra_patterns": [
|
| 615 |
+
(r'cursor\.execute|\.raw\s*\(', {
|
| 616 |
+
"title": "OWASP A03:2021 - Injection Violation",
|
| 617 |
+
"severity": "CRITICAL",
|
| 618 |
+
"category": "compliance",
|
| 619 |
+
"description": "Raw database query execution violates OWASP A03:2021 Injection guidelines",
|
| 620 |
+
"remediation": "Use parameterized queries per OWASP A03:2021 recommendations",
|
| 621 |
+
"owasp": "A03:2021",
|
| 622 |
+
"cwe": "CWE-89",
|
| 623 |
+
}),
|
| 624 |
+
(r'verify\s*=\s*False|CERT_NONE|_create_unverified_context', {
|
| 625 |
+
"title": "OWASP A02:2021 - Cryptographic Failure",
|
| 626 |
+
"severity": "HIGH",
|
| 627 |
+
"category": "compliance",
|
| 628 |
+
"description": "SSL verification disabled violates OWASP A02:2021 Cryptographic Failures",
|
| 629 |
+
"remediation": "Enable certificate verification per OWASP A02:2021",
|
| 630 |
+
"owasp": "A02:2021",
|
| 631 |
+
"cwe": "CWE-295",
|
| 632 |
+
}),
|
| 633 |
+
(r'(?:password|api_key|secret)\s*=\s*["\'][^"\']+["\']', {
|
| 634 |
+
"title": "OWASP A07:2021 - Authentication Failure",
|
| 635 |
+
"severity": "HIGH",
|
| 636 |
+
"category": "compliance",
|
| 637 |
+
"description": "Hardcoded credentials violate OWASP A07:2021 Identification and Authentication Failures",
|
| 638 |
+
"remediation": "Use secrets management per OWASP A07:2021",
|
| 639 |
+
"owasp": "A07:2021",
|
| 640 |
+
"cwe": "CWE-798",
|
| 641 |
+
}),
|
| 642 |
+
(r'@app\.route.*methods', {
|
| 643 |
+
"title": "OWASP A01:2021 - Broken Access Control Review",
|
| 644 |
+
"severity": "MEDIUM",
|
| 645 |
+
"category": "compliance",
|
| 646 |
+
"description": "HTTP endpoint requires access control review per OWASP A01:2021",
|
| 647 |
+
"remediation": "Implement authentication and authorization on all endpoints",
|
| 648 |
+
"owasp": "A01:2021",
|
| 649 |
+
}),
|
| 650 |
+
],
|
| 651 |
+
},
|
| 652 |
+
}
|
| 653 |
+
|
| 654 |
+
def __init__(self, config: LLMConfig):
|
| 655 |
+
super().__init__(config)
|
| 656 |
+
self._pattern_cache = {}
|
| 657 |
+
|
| 658 |
+
async def complete(self, messages: List[Dict[str, str]], **kwargs) -> str:
|
| 659 |
+
"""Analyze the code content in messages and return pattern-matched findings as JSON.
|
| 660 |
+
|
| 661 |
+
Detects which agent is calling based on the system prompt and returns
|
| 662 |
+
differentiated findings specific to that agent's specialty.
|
| 663 |
+
"""
|
| 664 |
+
# Extract code content and detect calling agent
|
| 665 |
+
code_content = ""
|
| 666 |
+
calling_agent = None
|
| 667 |
+
|
| 668 |
+
for msg in messages:
|
| 669 |
+
content = msg.get("content", "")
|
| 670 |
+
code_content += content + "\n"
|
| 671 |
+
|
| 672 |
+
# Detect which agent is calling by examining system prompt
|
| 673 |
+
if msg.get("role") == "system":
|
| 674 |
+
content_lower = content.lower()
|
| 675 |
+
for agent_name in self.AGENT_PATTERNS:
|
| 676 |
+
if agent_name.lower().replace("agent", "") in content_lower or agent_name.lower() in content_lower:
|
| 677 |
+
calling_agent = agent_name
|
| 678 |
+
break
|
| 679 |
+
# Additional keyword-based detection
|
| 680 |
+
if not calling_agent:
|
| 681 |
+
if "vulnerability" in content_lower and "detect" in content_lower:
|
| 682 |
+
calling_agent = "VulnAgent"
|
| 683 |
+
elif "threat" in content_lower and ("model" in content_lower or "attack" in content_lower):
|
| 684 |
+
calling_agent = "ThreatAgent"
|
| 685 |
+
elif "recon" in content_lower or "information disclosure" in content_lower:
|
| 686 |
+
calling_agent = "ReconAgent"
|
| 687 |
+
elif "compliance" in content_lower or "owasp" in content_lower:
|
| 688 |
+
calling_agent = "ComplianceAgent"
|
| 689 |
+
|
| 690 |
+
# Get base findings from shared patterns
|
| 691 |
+
findings = self._match_patterns(code_content)
|
| 692 |
+
|
| 693 |
+
# Add agent-specific findings (these are DIFFERENT from SAST findings)
|
| 694 |
+
if calling_agent and calling_agent in self.AGENT_PATTERNS:
|
| 695 |
+
agent_info = self.AGENT_PATTERNS[calling_agent]
|
| 696 |
+
for pattern_str, info in agent_info["extra_patterns"]:
|
| 697 |
+
try:
|
| 698 |
+
for match in re.finditer(pattern_str, code_content, re.IGNORECASE | re.DOTALL):
|
| 699 |
+
line_num = code_content[:match.start()].count("\n") + 1
|
| 700 |
+
lines = code_content.split("\n")
|
| 701 |
+
line_content = lines[line_num - 1].strip() if line_num <= len(lines) else ""
|
| 702 |
+
|
| 703 |
+
# Dedup by title+line within this agent
|
| 704 |
+
dedup_key = f"{info['title']}:{line_num}"
|
| 705 |
+
existing_keys = {f"{f['title']}:{f.get('line', 0)}" for f in findings}
|
| 706 |
+
if dedup_key in existing_keys:
|
| 707 |
+
continue
|
| 708 |
+
|
| 709 |
+
finding = {
|
| 710 |
+
**info,
|
| 711 |
+
"line": line_num,
|
| 712 |
+
"code_snippet": line_content[:200],
|
| 713 |
+
"source": calling_agent,
|
| 714 |
+
"confidence": 0.75,
|
| 715 |
+
}
|
| 716 |
+
findings.append(finding)
|
| 717 |
+
except re.error:
|
| 718 |
+
continue
|
| 719 |
+
|
| 720 |
+
# Build a realistic LLM-like response
|
| 721 |
+
result = {
|
| 722 |
+
"findings": findings,
|
| 723 |
+
"summary": f"Analysis complete. Found {len(findings)} potential security issues.",
|
| 724 |
+
"confidence": 0.85 if findings else 0.95,
|
| 725 |
+
}
|
| 726 |
+
|
| 727 |
+
return json.dumps(result, indent=2)
|
| 728 |
+
|
| 729 |
+
def _match_patterns(self, code: str) -> List[Dict[str, Any]]:
|
| 730 |
+
"""Match code against vulnerability patterns and return findings."""
|
| 731 |
+
findings = []
|
| 732 |
+
seen_titles = set()
|
| 733 |
+
|
| 734 |
+
lines = code.split("\n")
|
| 735 |
+
for pattern_info in self.VULNERABILITY_PATTERNS:
|
| 736 |
+
pattern = pattern_info[0]
|
| 737 |
+
info = pattern_info[1]
|
| 738 |
+
|
| 739 |
+
try:
|
| 740 |
+
for match in re.finditer(pattern, code, re.IGNORECASE | re.DOTALL):
|
| 741 |
+
# Find line number
|
| 742 |
+
line_num = code[:match.start()].count("\n") + 1
|
| 743 |
+
line_content = lines[line_num - 1].strip() if line_num <= len(lines) else ""
|
| 744 |
+
|
| 745 |
+
# Avoid duplicate findings of same type on adjacent lines
|
| 746 |
+
dedup_key = f"{info['title']}:{line_num}"
|
| 747 |
+
if dedup_key in seen_titles:
|
| 748 |
+
continue
|
| 749 |
+
seen_titles.add(dedup_key)
|
| 750 |
+
|
| 751 |
+
finding = {
|
| 752 |
+
**info,
|
| 753 |
+
"line": line_num,
|
| 754 |
+
"code_snippet": line_content[:200],
|
| 755 |
+
"source": "mock_provider",
|
| 756 |
+
"confidence": 0.8,
|
| 757 |
+
}
|
| 758 |
+
findings.append(finding)
|
| 759 |
+
except re.error:
|
| 760 |
+
continue
|
| 761 |
+
|
| 762 |
+
return findings
|
| 763 |
+
|
| 764 |
+
async def complete_json(self, messages: List[Dict[str, str]], **kwargs) -> Dict[str, Any]:
|
| 765 |
+
"""Return parsed JSON with fallback handling."""
|
| 766 |
+
raw = await self.complete(messages, **kwargs)
|
| 767 |
+
if not raw:
|
| 768 |
+
return {}
|
| 769 |
+
# Mock provider always returns valid JSON, but use parser for consistency
|
| 770 |
+
return LLMResponseParser.parse(raw)
|
| 771 |
+
|
| 772 |
+
|
| 773 |
+
def create_llm_provider(config: LLMConfig) -> BaseLLMProvider:
|
| 774 |
+
"""Factory function to create the appropriate LLM provider.
|
| 775 |
+
|
| 776 |
+
Args:
|
| 777 |
+
config: LLM configuration.
|
| 778 |
+
|
| 779 |
+
Returns:
|
| 780 |
+
An instance of the appropriate LLM provider.
|
| 781 |
+
"""
|
| 782 |
+
providers = {
|
| 783 |
+
"mock": MockProvider,
|
| 784 |
+
"openai": OpenAIProvider,
|
| 785 |
+
"anthropic": AnthropicProvider,
|
| 786 |
+
"ollama": OllamaProvider,
|
| 787 |
+
}
|
| 788 |
+
|
| 789 |
+
provider_class = providers.get(config.provider.lower())
|
| 790 |
+
if provider_class is None:
|
| 791 |
+
logger.warning(f"Unknown provider '{config.provider}', falling back to mock")
|
| 792 |
+
provider_class = MockProvider
|
| 793 |
+
|
| 794 |
+
return provider_class(config)
|
shield_agents/orchestrator.py
ADDED
|
@@ -0,0 +1,382 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Orchestrator for Shield Agents.
|
| 3 |
+
|
| 4 |
+
Central coordinator that manages all agents, scanners, deduplication,
|
| 5 |
+
caching, and shieldignore filtering. Provides a unified interface for
|
| 6 |
+
running security scans.
|
| 7 |
+
"""
|
| 8 |
+
|
| 9 |
+
import asyncio
|
| 10 |
+
import json
|
| 11 |
+
import logging
|
| 12 |
+
import time
|
| 13 |
+
from pathlib import Path
|
| 14 |
+
from typing import Any, Dict, List, Optional
|
| 15 |
+
|
| 16 |
+
from .config import ShieldConfig
|
| 17 |
+
from .llm import create_llm_provider
|
| 18 |
+
from .agents import VulnAgent, ThreatAgent, ReconAgent, ComplianceAgent, ResponseAgent, AutoFixAgent
|
| 19 |
+
from .scanners import SASTScanner, SecretsScanner
|
| 20 |
+
from .deduplication import FindingDeduplicator
|
| 21 |
+
from .cache import ScanCache
|
| 22 |
+
from .shieldignore import ShieldIgnore
|
| 23 |
+
from .report import ReportGenerator
|
| 24 |
+
from .utils.helpers import find_files, read_file_safe, calculate_risk_score
|
| 25 |
+
|
| 26 |
+
logger = logging.getLogger("shield_agents.orchestrator")
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
class ScanResult:
|
| 30 |
+
"""Result of a security scan."""
|
| 31 |
+
|
| 32 |
+
def __init__(self):
|
| 33 |
+
self.findings: List[Dict[str, Any]] = []
|
| 34 |
+
self.deduplicated_findings: List[Dict[str, Any]] = []
|
| 35 |
+
self.filtered_findings: List[Dict[str, Any]] = []
|
| 36 |
+
self.fixes: List[Dict[str, Any]] = []
|
| 37 |
+
self.risk_score: int = 0
|
| 38 |
+
self.scan_duration: float = 0.0
|
| 39 |
+
self.files_scanned: int = 0
|
| 40 |
+
self.stats: Dict[str, Any] = {}
|
| 41 |
+
self.report_files: Dict[str, str] = {}
|
| 42 |
+
|
| 43 |
+
def to_dict(self) -> Dict[str, Any]:
|
| 44 |
+
"""Convert to dictionary."""
|
| 45 |
+
return {
|
| 46 |
+
"total_findings": len(self.filtered_findings),
|
| 47 |
+
"risk_score": self.risk_score,
|
| 48 |
+
"scan_duration": round(self.scan_duration, 2),
|
| 49 |
+
"files_scanned": self.files_scanned,
|
| 50 |
+
"findings": self.filtered_findings,
|
| 51 |
+
"fixes": self.fixes,
|
| 52 |
+
"stats": self.stats,
|
| 53 |
+
"report_files": self.report_files,
|
| 54 |
+
}
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
class Orchestrator:
|
| 58 |
+
"""Central orchestrator for Shield Agents security scans.
|
| 59 |
+
|
| 60 |
+
Coordinates:
|
| 61 |
+
- Static analysis (SAST + Secrets scanning)
|
| 62 |
+
- AI agent analysis (Vuln, Threat, Recon, Compliance)
|
| 63 |
+
- Auto-fix generation
|
| 64 |
+
- Deduplication of findings
|
| 65 |
+
- .shieldignore filtering
|
| 66 |
+
- Caching / incremental scanning
|
| 67 |
+
- Report generation (HTML, SARIF, JSON)
|
| 68 |
+
"""
|
| 69 |
+
|
| 70 |
+
def __init__(self, config: Optional[ShieldConfig] = None):
|
| 71 |
+
"""Initialize the orchestrator.
|
| 72 |
+
|
| 73 |
+
Args:
|
| 74 |
+
config: Shield Agents configuration. Uses defaults if not provided.
|
| 75 |
+
"""
|
| 76 |
+
self.config = config or ShieldConfig()
|
| 77 |
+
|
| 78 |
+
# Initialize components
|
| 79 |
+
self.llm = create_llm_provider(self.config.llm)
|
| 80 |
+
self.sast_scanner = SASTScanner(self.config)
|
| 81 |
+
self.secrets_scanner = SecretsScanner(self.config)
|
| 82 |
+
self.deduplicator = FindingDeduplicator(
|
| 83 |
+
similarity_threshold=self.config.deduplication.similarity_threshold,
|
| 84 |
+
merge_sources=self.config.deduplication.merge_sources,
|
| 85 |
+
)
|
| 86 |
+
self.cache = ScanCache(
|
| 87 |
+
cache_dir=self.config.cache.cache_dir,
|
| 88 |
+
max_age_days=self.config.cache.max_cache_age_days,
|
| 89 |
+
)
|
| 90 |
+
self.shieldignore = ShieldIgnore(self.config.target_path)
|
| 91 |
+
self.report_generator = ReportGenerator(self.config)
|
| 92 |
+
|
| 93 |
+
# Initialize agents
|
| 94 |
+
self.agents = {}
|
| 95 |
+
if self.config.agents.vuln_agent:
|
| 96 |
+
self.agents["vuln"] = VulnAgent(self.config, self.llm)
|
| 97 |
+
if self.config.agents.threat_agent:
|
| 98 |
+
self.agents["threat"] = ThreatAgent(self.config, self.llm)
|
| 99 |
+
if self.config.agents.recon_agent:
|
| 100 |
+
self.agents["recon"] = ReconAgent(self.config, self.llm)
|
| 101 |
+
if self.config.agents.compliance_agent:
|
| 102 |
+
self.agents["compliance"] = ComplianceAgent(self.config, self.llm)
|
| 103 |
+
if self.config.agents.response_agent:
|
| 104 |
+
self.agents["response"] = ResponseAgent(self.config, self.llm)
|
| 105 |
+
if self.config.agents.autofix_agent:
|
| 106 |
+
self.agents["autofix"] = AutoFixAgent(self.config, self.llm)
|
| 107 |
+
|
| 108 |
+
async def scan(
|
| 109 |
+
self,
|
| 110 |
+
target_path: str,
|
| 111 |
+
full_scan: bool = False,
|
| 112 |
+
generate_report: bool = True,
|
| 113 |
+
generate_fixes: bool = True,
|
| 114 |
+
) -> ScanResult:
|
| 115 |
+
"""Run a complete security scan.
|
| 116 |
+
|
| 117 |
+
Args:
|
| 118 |
+
target_path: Path to scan.
|
| 119 |
+
full_scan: If True, ignore cache and scan all files.
|
| 120 |
+
generate_report: If True, generate output reports.
|
| 121 |
+
generate_fixes: If True, generate auto-fix suggestions.
|
| 122 |
+
|
| 123 |
+
Returns:
|
| 124 |
+
ScanResult with all findings and metadata.
|
| 125 |
+
"""
|
| 126 |
+
start_time = time.time()
|
| 127 |
+
result = ScanResult()
|
| 128 |
+
|
| 129 |
+
# Update target path
|
| 130 |
+
self.config.target_path = target_path
|
| 131 |
+
|
| 132 |
+
# Load .shieldignore rules
|
| 133 |
+
if self.config.shieldignore.enabled:
|
| 134 |
+
self.shieldignore = ShieldIgnore(target_path)
|
| 135 |
+
self.shieldignore.load()
|
| 136 |
+
|
| 137 |
+
# Clean up expired cache entries
|
| 138 |
+
if self.config.cache.enabled:
|
| 139 |
+
self.cache.cleanup_expired()
|
| 140 |
+
|
| 141 |
+
# Find source files to scan
|
| 142 |
+
source_extensions = [
|
| 143 |
+
".py", ".js", ".ts", ".jsx", ".tsx", ".java", ".go", ".rb",
|
| 144 |
+
".php", ".c", ".cpp", ".cs", ".rs", ".swift", ".kt",
|
| 145 |
+
".html", ".htm", ".xml", ".yaml", ".yml", ".json", ".sql",
|
| 146 |
+
".env", ".cfg", ".conf", ".toml", ".properties",
|
| 147 |
+
]
|
| 148 |
+
|
| 149 |
+
# If target is a single file, scan it directly without directory traversal
|
| 150 |
+
target_path_obj = Path(target_path).resolve()
|
| 151 |
+
content_dirs_to_exclude = [] # Initialize for later reference
|
| 152 |
+
|
| 153 |
+
if target_path_obj.is_file():
|
| 154 |
+
all_files = [str(target_path_obj)]
|
| 155 |
+
else:
|
| 156 |
+
# Auto-exclude content dirs (tests, benchmarks, examples) only when they
|
| 157 |
+
# are subdirectories - NOT when the user explicitly targets them
|
| 158 |
+
target_name = target_path_obj.name
|
| 159 |
+
|
| 160 |
+
content_dirs_to_exclude = [
|
| 161 |
+
d for d in self.config.scanner.excluded_content_dirs
|
| 162 |
+
if d != target_name # Don't exclude the target itself
|
| 163 |
+
]
|
| 164 |
+
|
| 165 |
+
excluded_dirs = (
|
| 166 |
+
self.config.scanner.excluded_dirs +
|
| 167 |
+
content_dirs_to_exclude
|
| 168 |
+
)
|
| 169 |
+
|
| 170 |
+
all_files = find_files(
|
| 171 |
+
target_path,
|
| 172 |
+
extensions=source_extensions,
|
| 173 |
+
exclude_dirs=excluded_dirs,
|
| 174 |
+
)
|
| 175 |
+
|
| 176 |
+
# Filter by .shieldignore path rules
|
| 177 |
+
if self.shieldignore.rules and self.shieldignore._loaded:
|
| 178 |
+
filtered = []
|
| 179 |
+
for fp in all_files:
|
| 180 |
+
dummy_finding = {"file": fp, "title": "", "category": "", "severity": "MEDIUM"}
|
| 181 |
+
if not self.shieldignore.should_ignore(dummy_finding):
|
| 182 |
+
filtered.append(fp)
|
| 183 |
+
all_files = filtered
|
| 184 |
+
|
| 185 |
+
# Filter out files in excluded content directories (tests, benchmarks, examples)
|
| 186 |
+
# But only subdirectories, not the target itself
|
| 187 |
+
if content_dirs_to_exclude:
|
| 188 |
+
filtered_files = []
|
| 189 |
+
for fp in all_files:
|
| 190 |
+
parts = Path(fp).parts
|
| 191 |
+
# Don't exclude if this is the target directory itself
|
| 192 |
+
if not any(exc in parts for exc in content_dirs_to_exclude):
|
| 193 |
+
filtered_files.append(fp)
|
| 194 |
+
all_files = filtered_files
|
| 195 |
+
|
| 196 |
+
# Filter out excluded filenames
|
| 197 |
+
if self.config.scanner.excluded_filenames:
|
| 198 |
+
filtered_files = []
|
| 199 |
+
for fp in all_files:
|
| 200 |
+
filename = Path(fp).name
|
| 201 |
+
if not any(filename.startswith(exc) or filename.endswith(exc) for exc in self.config.scanner.excluded_filenames):
|
| 202 |
+
filtered_files.append(fp)
|
| 203 |
+
all_files = filtered_files
|
| 204 |
+
|
| 205 |
+
# Determine which files need re-scanning (incremental)
|
| 206 |
+
if self.config.cache.enabled and self.config.cache.incremental and not full_scan:
|
| 207 |
+
changed_files = self.cache.get_changed_files(all_files)
|
| 208 |
+
unchanged_files = self.cache.get_unchanged_files(all_files)
|
| 209 |
+
|
| 210 |
+
logger.info(f"Incremental scan: {len(changed_files)} changed, {len(unchanged_files)} unchanged")
|
| 211 |
+
|
| 212 |
+
# Get cached findings for unchanged files
|
| 213 |
+
cached_findings = []
|
| 214 |
+
for fp in unchanged_files:
|
| 215 |
+
cached_findings.extend(self.cache.get_cached_findings(fp))
|
| 216 |
+
|
| 217 |
+
files_to_scan = changed_files
|
| 218 |
+
else:
|
| 219 |
+
files_to_scan = all_files
|
| 220 |
+
cached_findings = []
|
| 221 |
+
|
| 222 |
+
result.files_scanned = len(files_to_scan)
|
| 223 |
+
|
| 224 |
+
# Phase 1: Static analysis (SAST + Secrets)
|
| 225 |
+
all_findings = list(cached_findings)
|
| 226 |
+
|
| 227 |
+
# SAST scanning
|
| 228 |
+
if self.config.scanner.sast_enabled:
|
| 229 |
+
sast_findings = self._run_sast_scan(files_to_scan)
|
| 230 |
+
all_findings.extend(sast_findings)
|
| 231 |
+
|
| 232 |
+
# Secrets scanning
|
| 233 |
+
if self.config.scanner.secrets_enabled:
|
| 234 |
+
secrets_findings = self._run_secrets_scan(files_to_scan)
|
| 235 |
+
all_findings.extend(secrets_findings)
|
| 236 |
+
|
| 237 |
+
# Phase 2: AI agent analysis
|
| 238 |
+
agent_findings = await self._run_agent_analysis(files_to_scan)
|
| 239 |
+
all_findings.extend(agent_findings)
|
| 240 |
+
|
| 241 |
+
# Phase 3: Deduplication
|
| 242 |
+
if self.config.deduplication.enabled:
|
| 243 |
+
result.deduplicated_findings = self.deduplicator.deduplicate(all_findings)
|
| 244 |
+
result.stats["deduplication"] = self.deduplicator.get_stats()
|
| 245 |
+
else:
|
| 246 |
+
result.deduplicated_findings = all_findings
|
| 247 |
+
|
| 248 |
+
# Phase 4: .shieldignore filtering
|
| 249 |
+
if self.config.shieldignore.enabled:
|
| 250 |
+
result.filtered_findings = self.shieldignore.filter_findings(result.deduplicated_findings)
|
| 251 |
+
result.stats["shieldignore"] = self.shieldignore.get_stats()
|
| 252 |
+
else:
|
| 253 |
+
result.filtered_findings = result.deduplicated_findings
|
| 254 |
+
|
| 255 |
+
# Phase 5: Auto-fix generation
|
| 256 |
+
if generate_fixes and "autofix" in self.agents:
|
| 257 |
+
findings_json = json.dumps(result.filtered_findings[:50]) # Limit for performance
|
| 258 |
+
# Get code content for context
|
| 259 |
+
code_content = self._gather_code_content(files_to_scan[:20])
|
| 260 |
+
result.fixes = await self.agents["autofix"].analyze(
|
| 261 |
+
findings_json, code_content=code_content
|
| 262 |
+
)
|
| 263 |
+
|
| 264 |
+
# Phase 6: Response agent (risk assessment and recommendations)
|
| 265 |
+
if "response" in self.agents:
|
| 266 |
+
response_findings = await self.agents["response"].analyze(
|
| 267 |
+
json.dumps(result.filtered_findings[:30])
|
| 268 |
+
)
|
| 269 |
+
result.stats["response_recommendations"] = len(response_findings)
|
| 270 |
+
|
| 271 |
+
# Calculate risk score
|
| 272 |
+
result.risk_score = calculate_risk_score(result.filtered_findings)
|
| 273 |
+
result.findings = all_findings
|
| 274 |
+
|
| 275 |
+
# Update cache
|
| 276 |
+
if self.config.cache.enabled:
|
| 277 |
+
self._update_cache(files_to_scan, all_findings)
|
| 278 |
+
self.cache.save()
|
| 279 |
+
|
| 280 |
+
# Generate reports
|
| 281 |
+
if generate_report:
|
| 282 |
+
scan_stats = {
|
| 283 |
+
"files_scanned": result.files_scanned,
|
| 284 |
+
"total_files": len(all_files),
|
| 285 |
+
"scan_duration": time.time() - start_time,
|
| 286 |
+
"cache_enabled": self.config.cache.enabled,
|
| 287 |
+
"dedup_enabled": self.config.deduplication.enabled,
|
| 288 |
+
"shieldignore_enabled": self.config.shieldignore.enabled,
|
| 289 |
+
}
|
| 290 |
+
result.report_files = self.report_generator.generate_report(
|
| 291 |
+
findings=result.filtered_findings,
|
| 292 |
+
target_path=target_path,
|
| 293 |
+
output_dir=self.config.report.output_dir,
|
| 294 |
+
formats=self.config.report.formats,
|
| 295 |
+
scan_stats=scan_stats,
|
| 296 |
+
)
|
| 297 |
+
|
| 298 |
+
result.scan_duration = time.time() - start_time
|
| 299 |
+
result.stats["total_input_findings"] = len(all_findings)
|
| 300 |
+
result.stats["after_dedup"] = len(result.deduplicated_findings)
|
| 301 |
+
result.stats["after_filter"] = len(result.filtered_findings)
|
| 302 |
+
|
| 303 |
+
logger.info(
|
| 304 |
+
f"Scan complete: {len(all_findings)} findings → "
|
| 305 |
+
f"{len(result.filtered_findings)} after dedup+filter, "
|
| 306 |
+
f"risk score: {result.risk_score}/100, "
|
| 307 |
+
f"duration: {result.scan_duration:.2f}s"
|
| 308 |
+
)
|
| 309 |
+
|
| 310 |
+
return result
|
| 311 |
+
|
| 312 |
+
def _run_sast_scan(self, files: List[str]) -> List[Dict[str, Any]]:
|
| 313 |
+
"""Run SAST scanner on files."""
|
| 314 |
+
all_findings = []
|
| 315 |
+
for file_path in files:
|
| 316 |
+
findings = self.sast_scanner.scan_file(file_path)
|
| 317 |
+
all_findings.extend(findings)
|
| 318 |
+
return all_findings
|
| 319 |
+
|
| 320 |
+
def _run_secrets_scan(self, files: List[str]) -> List[Dict[str, Any]]:
|
| 321 |
+
"""Run Secrets scanner on files."""
|
| 322 |
+
all_findings = []
|
| 323 |
+
for file_path in files:
|
| 324 |
+
findings = self.secrets_scanner.scan_file(file_path)
|
| 325 |
+
all_findings.extend(findings)
|
| 326 |
+
return all_findings
|
| 327 |
+
|
| 328 |
+
async def _run_agent_analysis(self, files: List[str]) -> List[Dict[str, Any]]:
|
| 329 |
+
"""Run AI agent analysis on files."""
|
| 330 |
+
all_findings = []
|
| 331 |
+
|
| 332 |
+
# Limit the number of files sent to LLM for performance
|
| 333 |
+
max_files_for_llm = 10
|
| 334 |
+
files_for_llm = files[:max_files_for_llm]
|
| 335 |
+
|
| 336 |
+
for file_path in files_for_llm:
|
| 337 |
+
content = read_file_safe(file_path)
|
| 338 |
+
if not content:
|
| 339 |
+
continue
|
| 340 |
+
|
| 341 |
+
# Run enabled agents concurrently for each file
|
| 342 |
+
tasks = []
|
| 343 |
+
for name, agent in self.agents.items():
|
| 344 |
+
if name in ("response", "autofix"):
|
| 345 |
+
continue # These run later with aggregated findings
|
| 346 |
+
tasks.append(agent.analyze(content, file_path=file_path))
|
| 347 |
+
|
| 348 |
+
if tasks:
|
| 349 |
+
results = await asyncio.gather(*tasks, return_exceptions=True)
|
| 350 |
+
for r in results:
|
| 351 |
+
if isinstance(r, list):
|
| 352 |
+
all_findings.extend(r)
|
| 353 |
+
elif isinstance(r, Exception):
|
| 354 |
+
logger.warning(f"Agent error: {r}")
|
| 355 |
+
|
| 356 |
+
return all_findings
|
| 357 |
+
|
| 358 |
+
def _gather_code_content(self, files: List[str], max_files: int = 20) -> str:
|
| 359 |
+
"""Gather code content from multiple files for context."""
|
| 360 |
+
content_parts = []
|
| 361 |
+
for fp in files[:max_files]:
|
| 362 |
+
content = read_file_safe(fp)
|
| 363 |
+
if content:
|
| 364 |
+
# Truncate very long files
|
| 365 |
+
if len(content) > 5000:
|
| 366 |
+
content = content[:5000] + "\n... (truncated)"
|
| 367 |
+
content_parts.append(f"--- {fp} ---\n{content}\n")
|
| 368 |
+
return "\n".join(content_parts)
|
| 369 |
+
|
| 370 |
+
def _update_cache(self, files: List[str], findings: List[Dict[str, Any]]) -> None:
|
| 371 |
+
"""Update cache with scan results per file."""
|
| 372 |
+
# Group findings by file
|
| 373 |
+
findings_by_file: Dict[str, List[Dict[str, Any]]] = {}
|
| 374 |
+
for f in findings:
|
| 375 |
+
file_path = f.get("file", "")
|
| 376 |
+
if file_path:
|
| 377 |
+
findings_by_file.setdefault(file_path, []).append(f)
|
| 378 |
+
|
| 379 |
+
# Update cache for each scanned file
|
| 380 |
+
for file_path in files:
|
| 381 |
+
file_findings = findings_by_file.get(file_path, [])
|
| 382 |
+
self.cache.update_file(file_path, file_findings)
|
shield_agents/report/__init__.py
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Report generation for Shield Agents."""
|
| 2 |
+
|
| 3 |
+
from .generator import ReportGenerator
|
| 4 |
+
from .sarif import SARIFGenerator
|
| 5 |
+
|
| 6 |
+
__all__ = ["ReportGenerator", "SARIFGenerator"]
|
shield_agents/report/generator.py
ADDED
|
@@ -0,0 +1,352 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Report Generator for Shield Agents.
|
| 3 |
+
|
| 4 |
+
Generates HTML reports with interactive visualizations, severity breakdowns,
|
| 5 |
+
and detailed findings. Also supports JSON and plain text output.
|
| 6 |
+
"""
|
| 7 |
+
|
| 8 |
+
import json
|
| 9 |
+
import logging
|
| 10 |
+
import os
|
| 11 |
+
from datetime import datetime
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
from typing import Any, Dict, List, Optional
|
| 14 |
+
|
| 15 |
+
from .sarif import SARIFGenerator
|
| 16 |
+
from ..utils.helpers import calculate_risk_score
|
| 17 |
+
|
| 18 |
+
logger = logging.getLogger("shield_agents.report")
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
class ReportGenerator:
|
| 22 |
+
"""Generate security scan reports in multiple formats.
|
| 23 |
+
|
| 24 |
+
Supported formats:
|
| 25 |
+
- HTML: Interactive web report with charts and filtering
|
| 26 |
+
- SARIF: For GitHub Security tab integration
|
| 27 |
+
- JSON: Machine-readable structured data
|
| 28 |
+
"""
|
| 29 |
+
|
| 30 |
+
def __init__(self, config=None):
|
| 31 |
+
self.config = config
|
| 32 |
+
self.sarif_generator = SARIFGenerator(config)
|
| 33 |
+
|
| 34 |
+
def generate_report(
|
| 35 |
+
self,
|
| 36 |
+
findings: List[Dict[str, Any]],
|
| 37 |
+
target_path: str = ".",
|
| 38 |
+
output_dir: str = "./shield-reports",
|
| 39 |
+
formats: Optional[List[str]] = None,
|
| 40 |
+
scan_stats: Optional[Dict[str, Any]] = None,
|
| 41 |
+
) -> Dict[str, str]:
|
| 42 |
+
"""Generate reports in specified formats.
|
| 43 |
+
|
| 44 |
+
Args:
|
| 45 |
+
findings: List of finding dictionaries.
|
| 46 |
+
target_path: Path that was scanned.
|
| 47 |
+
output_dir: Directory to save reports.
|
| 48 |
+
formats: List of formats to generate (html, sarif, json).
|
| 49 |
+
scan_stats: Optional scan statistics.
|
| 50 |
+
|
| 51 |
+
Returns:
|
| 52 |
+
Dictionary of format -> file path.
|
| 53 |
+
"""
|
| 54 |
+
if formats is None:
|
| 55 |
+
formats = ["html", "sarif", "json"]
|
| 56 |
+
|
| 57 |
+
# Ensure output directory exists
|
| 58 |
+
Path(output_dir).mkdir(parents=True, exist_ok=True)
|
| 59 |
+
|
| 60 |
+
# Calculate summary stats
|
| 61 |
+
risk_score = calculate_risk_score(findings)
|
| 62 |
+
summary = self._build_summary(findings, risk_score, scan_stats)
|
| 63 |
+
|
| 64 |
+
output_files = {}
|
| 65 |
+
|
| 66 |
+
for fmt in formats:
|
| 67 |
+
try:
|
| 68 |
+
if fmt == "html":
|
| 69 |
+
path = self._generate_html(findings, summary, target_path, output_dir)
|
| 70 |
+
output_files["html"] = path
|
| 71 |
+
elif fmt == "sarif":
|
| 72 |
+
path = self._generate_sarif(findings, target_path, output_dir)
|
| 73 |
+
output_files["sarif"] = path
|
| 74 |
+
elif fmt == "json":
|
| 75 |
+
path = self._generate_json(findings, summary, target_path, output_dir)
|
| 76 |
+
output_files["json"] = path
|
| 77 |
+
else:
|
| 78 |
+
logger.warning(f"Unknown report format: {fmt}")
|
| 79 |
+
except Exception as e:
|
| 80 |
+
logger.error(f"Failed to generate {fmt} report: {e}")
|
| 81 |
+
|
| 82 |
+
logger.info(f"Generated reports: {list(output_files.keys())}")
|
| 83 |
+
return output_files
|
| 84 |
+
|
| 85 |
+
def _build_summary(
|
| 86 |
+
self,
|
| 87 |
+
findings: List[Dict[str, Any]],
|
| 88 |
+
risk_score: int,
|
| 89 |
+
scan_stats: Optional[Dict[str, Any]] = None,
|
| 90 |
+
) -> Dict[str, Any]:
|
| 91 |
+
"""Build summary statistics from findings."""
|
| 92 |
+
severity_counts = {"CRITICAL": 0, "HIGH": 0, "MEDIUM": 0, "LOW": 0, "INFO": 0}
|
| 93 |
+
category_counts: Dict[str, int] = {}
|
| 94 |
+
source_counts: Dict[str, int] = {}
|
| 95 |
+
file_counts: Dict[str, int] = {}
|
| 96 |
+
|
| 97 |
+
for f in findings:
|
| 98 |
+
sev = f.get("severity", "MEDIUM").upper()
|
| 99 |
+
severity_counts[sev] = severity_counts.get(sev, 0) + 1
|
| 100 |
+
|
| 101 |
+
cat = f.get("category", "unknown")
|
| 102 |
+
category_counts[cat] = category_counts.get(cat, 0) + 1
|
| 103 |
+
|
| 104 |
+
source = f.get("source", f.get("agent", "unknown"))
|
| 105 |
+
source_counts[source] = source_counts.get(source, 0) + 1
|
| 106 |
+
|
| 107 |
+
file = f.get("file", "unknown")
|
| 108 |
+
file_counts[file] = file_counts.get(file, 0) + 1
|
| 109 |
+
|
| 110 |
+
summary = {
|
| 111 |
+
"total_findings": len(findings),
|
| 112 |
+
"risk_score": risk_score,
|
| 113 |
+
"severity_counts": severity_counts,
|
| 114 |
+
"category_counts": category_counts,
|
| 115 |
+
"source_counts": source_counts,
|
| 116 |
+
"most_affected_files": sorted(file_counts.items(), key=lambda x: x[1], reverse=True)[:10],
|
| 117 |
+
"generated_at": datetime.now().isoformat(),
|
| 118 |
+
}
|
| 119 |
+
|
| 120 |
+
if scan_stats:
|
| 121 |
+
summary["scan_stats"] = scan_stats
|
| 122 |
+
|
| 123 |
+
return summary
|
| 124 |
+
|
| 125 |
+
def _generate_sarif(self, findings: List[Dict[str, Any]], target_path: str, output_dir: str) -> str:
|
| 126 |
+
"""Generate SARIF report."""
|
| 127 |
+
output_path = os.path.join(output_dir, "results.sarif")
|
| 128 |
+
return self.sarif_generator.save(findings, output_path, target_path)
|
| 129 |
+
|
| 130 |
+
def _generate_json(
|
| 131 |
+
self,
|
| 132 |
+
findings: List[Dict[str, Any]],
|
| 133 |
+
summary: Dict[str, Any],
|
| 134 |
+
target_path: str,
|
| 135 |
+
output_dir: str,
|
| 136 |
+
) -> str:
|
| 137 |
+
"""Generate JSON report."""
|
| 138 |
+
output_path = os.path.join(output_dir, "results.json")
|
| 139 |
+
report = {
|
| 140 |
+
"shield_agents_version": "2.0.0",
|
| 141 |
+
"target_path": target_path,
|
| 142 |
+
"summary": summary,
|
| 143 |
+
"findings": findings,
|
| 144 |
+
}
|
| 145 |
+
with open(output_path, "w") as f:
|
| 146 |
+
json.dump(report, f, indent=2)
|
| 147 |
+
return output_path
|
| 148 |
+
|
| 149 |
+
def _generate_html(
|
| 150 |
+
self,
|
| 151 |
+
findings: List[Dict[str, Any]],
|
| 152 |
+
summary: Dict[str, Any],
|
| 153 |
+
target_path: str,
|
| 154 |
+
output_dir: str,
|
| 155 |
+
) -> str:
|
| 156 |
+
"""Generate HTML report with interactive visualizations."""
|
| 157 |
+
output_path = os.path.join(output_dir, "report.html")
|
| 158 |
+
|
| 159 |
+
severity_colors = {
|
| 160 |
+
"CRITICAL": "#dc3545",
|
| 161 |
+
"HIGH": "#fd7e14",
|
| 162 |
+
"MEDIUM": "#ffc107",
|
| 163 |
+
"LOW": "#0dcaf0",
|
| 164 |
+
"INFO": "#6c757d",
|
| 165 |
+
}
|
| 166 |
+
|
| 167 |
+
# Risk level text
|
| 168 |
+
if summary["risk_score"] >= 75:
|
| 169 |
+
risk_level = "CRITICAL"
|
| 170 |
+
risk_color = "#dc3545"
|
| 171 |
+
elif summary["risk_score"] >= 50:
|
| 172 |
+
risk_level = "HIGH"
|
| 173 |
+
risk_color = "#fd7e14"
|
| 174 |
+
elif summary["risk_score"] >= 25:
|
| 175 |
+
risk_level = "MEDIUM"
|
| 176 |
+
risk_color = "#ffc107"
|
| 177 |
+
else:
|
| 178 |
+
risk_level = "LOW"
|
| 179 |
+
risk_color = "#0dcaf0"
|
| 180 |
+
|
| 181 |
+
# Build severity breakdown bars
|
| 182 |
+
total = max(summary["total_findings"], 1)
|
| 183 |
+
severity_bars = ""
|
| 184 |
+
for sev in ["CRITICAL", "HIGH", "MEDIUM", "LOW", "INFO"]:
|
| 185 |
+
count = summary["severity_counts"].get(sev, 0)
|
| 186 |
+
pct = (count / total) * 100
|
| 187 |
+
color = severity_colors.get(sev, "#6c757d")
|
| 188 |
+
severity_bars += f"""
|
| 189 |
+
<div class="severity-row">
|
| 190 |
+
<span class="severity-label">{sev}</span>
|
| 191 |
+
<div class="severity-bar-container">
|
| 192 |
+
<div class="severity-bar" style="width: {pct}%; background-color: {color};"></div>
|
| 193 |
+
</div>
|
| 194 |
+
<span class="severity-count">{count}</span>
|
| 195 |
+
</div>"""
|
| 196 |
+
|
| 197 |
+
# Build findings table
|
| 198 |
+
findings_rows = ""
|
| 199 |
+
for i, f in enumerate(findings[:200]): # Limit to 200 rows
|
| 200 |
+
sev = f.get("severity", "MEDIUM").upper()
|
| 201 |
+
color = severity_colors.get(sev, "#6c757d")
|
| 202 |
+
title = f.get("title", "Unknown").replace("<", "<").replace(">", ">")
|
| 203 |
+
desc = f.get("description", "").replace("<", "<").replace(">", ">")[:150]
|
| 204 |
+
file = f.get("file", "N/A").replace("<", "<").replace(">", ">")
|
| 205 |
+
line = f.get("line", "N/A")
|
| 206 |
+
source = f.get("source", f.get("agent", "unknown"))
|
| 207 |
+
remediation = f.get("remediation", "").replace("<", "<").replace(">", ">")[:200]
|
| 208 |
+
sources = ", ".join(f.get("sources", [source]))
|
| 209 |
+
cwe = f.get("cwe", "")
|
| 210 |
+
|
| 211 |
+
findings_rows += f"""
|
| 212 |
+
<tr>
|
| 213 |
+
<td><span class="badge" style="background-color: {color}">{sev}</span></td>
|
| 214 |
+
<td>{title}</td>
|
| 215 |
+
<td>{desc}...</td>
|
| 216 |
+
<td class="mono">{file}</td>
|
| 217 |
+
<td>{line}</td>
|
| 218 |
+
<td>{sources}</td>
|
| 219 |
+
<td>{cwe}</td>
|
| 220 |
+
<td>{remediation}...</td>
|
| 221 |
+
</tr>"""
|
| 222 |
+
|
| 223 |
+
# Category breakdown
|
| 224 |
+
category_items = ""
|
| 225 |
+
for cat, count in sorted(summary["category_counts"].items(), key=lambda x: x[1], reverse=True):
|
| 226 |
+
category_items += f'<li><strong>{cat}</strong>: {count}</li>'
|
| 227 |
+
|
| 228 |
+
# Most affected files
|
| 229 |
+
file_items = ""
|
| 230 |
+
for filepath, count in summary["most_affected_files"][:10]:
|
| 231 |
+
file_items += f'<li><code>{filepath}</code>: {count} findings</li>'
|
| 232 |
+
|
| 233 |
+
html = f"""<!DOCTYPE html>
|
| 234 |
+
<html lang="en">
|
| 235 |
+
<head>
|
| 236 |
+
<meta charset="UTF-8">
|
| 237 |
+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
| 238 |
+
<title>Shield Agents - Security Report</title>
|
| 239 |
+
<style>
|
| 240 |
+
* {{ margin: 0; padding: 0; box-sizing: border-box; }}
|
| 241 |
+
body {{ font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #0d1117; color: #c9d1d9; line-height: 1.6; }}
|
| 242 |
+
.container {{ max-width: 1400px; margin: 0 auto; padding: 20px; }}
|
| 243 |
+
.header {{ background: linear-gradient(135deg, #1a1b26 0%, #24283b 100%); border-radius: 12px; padding: 30px; margin-bottom: 24px; border: 1px solid #30363d; }}
|
| 244 |
+
.header h1 {{ font-size: 28px; color: #58a6ff; margin-bottom: 8px; }}
|
| 245 |
+
.header p {{ color: #8b949e; }}
|
| 246 |
+
.grid {{ display: grid; grid-template-columns: repeat(auto-fit, minmax(300px, 1fr)); gap: 20px; margin-bottom: 24px; }}
|
| 247 |
+
.card {{ background: #161b22; border-radius: 12px; padding: 24px; border: 1px solid #30363d; }}
|
| 248 |
+
.card h2 {{ font-size: 18px; color: #58a6ff; margin-bottom: 16px; }}
|
| 249 |
+
.risk-score {{ font-size: 64px; font-weight: bold; text-align: center; color: {risk_color}; }}
|
| 250 |
+
.risk-label {{ text-align: center; font-size: 18px; color: {risk_color}; margin-top: 8px; }}
|
| 251 |
+
.severity-row {{ display: flex; align-items: center; margin-bottom: 8px; }}
|
| 252 |
+
.severity-label {{ width: 80px; font-weight: 600; font-size: 13px; }}
|
| 253 |
+
.severity-bar-container {{ flex: 1; height: 20px; background: #21262d; border-radius: 4px; overflow: hidden; }}
|
| 254 |
+
.severity-bar {{ height: 100%; border-radius: 4px; transition: width 0.5s; }}
|
| 255 |
+
.severity-count {{ width: 40px; text-align: right; font-weight: 600; margin-left: 8px; }}
|
| 256 |
+
table {{ width: 100%; border-collapse: collapse; font-size: 13px; }}
|
| 257 |
+
th {{ background: #21262d; color: #8b949e; padding: 12px 8px; text-align: left; position: sticky; top: 0; }}
|
| 258 |
+
td {{ padding: 10px 8px; border-bottom: 1px solid #21262d; }}
|
| 259 |
+
tr:hover {{ background: #1c2128; }}
|
| 260 |
+
.badge {{ padding: 2px 8px; border-radius: 12px; font-size: 11px; font-weight: 600; color: white; }}
|
| 261 |
+
.mono {{ font-family: 'SF Mono', Monaco, monospace; font-size: 12px; }}
|
| 262 |
+
.stats {{ display: grid; grid-template-columns: repeat(4, 1fr); gap: 12px; margin-top: 16px; }}
|
| 263 |
+
.stat-item {{ text-align: center; }}
|
| 264 |
+
.stat-value {{ font-size: 24px; font-weight: bold; color: #58a6ff; }}
|
| 265 |
+
.stat-label {{ font-size: 12px; color: #8b949e; }}
|
| 266 |
+
.table-container {{ overflow-x: auto; max-height: 600px; overflow-y: auto; }}
|
| 267 |
+
.footer {{ text-align: center; padding: 20px; color: #484f58; font-size: 12px; }}
|
| 268 |
+
</style>
|
| 269 |
+
</head>
|
| 270 |
+
<body>
|
| 271 |
+
<div class="container">
|
| 272 |
+
<div class="header">
|
| 273 |
+
<h1>🛡 Shield Agents Security Report</h1>
|
| 274 |
+
<p>Target: <code>{target_path}</code> | Generated: {summary['generated_at']}</p>
|
| 275 |
+
</div>
|
| 276 |
+
|
| 277 |
+
<div class="grid">
|
| 278 |
+
<div class="card">
|
| 279 |
+
<h2>Risk Score</h2>
|
| 280 |
+
<div class="risk-score">{summary['risk_score']}</div>
|
| 281 |
+
<div class="risk-label">{risk_level} RISK</div>
|
| 282 |
+
<div class="stats">
|
| 283 |
+
<div class="stat-item">
|
| 284 |
+
<div class="stat-value">{summary['total_findings']}</div>
|
| 285 |
+
<div class="stat-label">Findings</div>
|
| 286 |
+
</div>
|
| 287 |
+
<div class="stat-item">
|
| 288 |
+
<div class="stat-value">{summary['severity_counts'].get('CRITICAL', 0)}</div>
|
| 289 |
+
<div class="stat-label">Critical</div>
|
| 290 |
+
</div>
|
| 291 |
+
<div class="stat-item">
|
| 292 |
+
<div class="stat-value">{summary['severity_counts'].get('HIGH', 0)}</div>
|
| 293 |
+
<div class="stat-label">High</div>
|
| 294 |
+
</div>
|
| 295 |
+
<div class="stat-item">
|
| 296 |
+
<div class="stat-value">{len(summary['category_counts'])}</div>
|
| 297 |
+
<div class="stat-label">Categories</div>
|
| 298 |
+
</div>
|
| 299 |
+
</div>
|
| 300 |
+
</div>
|
| 301 |
+
|
| 302 |
+
<div class="card">
|
| 303 |
+
<h2>Severity Distribution</h2>
|
| 304 |
+
{severity_bars}
|
| 305 |
+
</div>
|
| 306 |
+
|
| 307 |
+
<div class="card">
|
| 308 |
+
<h2>Categories</h2>
|
| 309 |
+
<ul style="list-style: none; padding: 0;">
|
| 310 |
+
{category_items}
|
| 311 |
+
</ul>
|
| 312 |
+
<h2 style="margin-top: 16px;">Most Affected Files</h2>
|
| 313 |
+
<ul style="list-style: none; padding: 0; font-size: 13px;">
|
| 314 |
+
{file_items}
|
| 315 |
+
</ul>
|
| 316 |
+
</div>
|
| 317 |
+
</div>
|
| 318 |
+
|
| 319 |
+
<div class="card">
|
| 320 |
+
<h2>Findings Detail ({summary['total_findings']} total)</h2>
|
| 321 |
+
<div class="table-container">
|
| 322 |
+
<table>
|
| 323 |
+
<thead>
|
| 324 |
+
<tr>
|
| 325 |
+
<th>Severity</th>
|
| 326 |
+
<th>Title</th>
|
| 327 |
+
<th>Description</th>
|
| 328 |
+
<th>File</th>
|
| 329 |
+
<th>Line</th>
|
| 330 |
+
<th>Source</th>
|
| 331 |
+
<th>CWE</th>
|
| 332 |
+
<th>Remediation</th>
|
| 333 |
+
</tr>
|
| 334 |
+
</thead>
|
| 335 |
+
<tbody>
|
| 336 |
+
{findings_rows}
|
| 337 |
+
</tbody>
|
| 338 |
+
</table>
|
| 339 |
+
</div>
|
| 340 |
+
</div>
|
| 341 |
+
|
| 342 |
+
<div class="footer">
|
| 343 |
+
Generated by Shield Agents v2.0.0
|
| 344 |
+
</div>
|
| 345 |
+
</div>
|
| 346 |
+
</body>
|
| 347 |
+
</html>"""
|
| 348 |
+
|
| 349 |
+
with open(output_path, "w", encoding="utf-8") as f:
|
| 350 |
+
f.write(html)
|
| 351 |
+
|
| 352 |
+
return output_path
|
shield_agents/report/sarif.py
ADDED
|
@@ -0,0 +1,240 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
SARIF (Static Analysis Results Interchange Format) Output for Shield Agents.
|
| 3 |
+
|
| 4 |
+
Generates SARIF 2.1.0 output for GitHub Security tab integration.
|
| 5 |
+
This is essential for GitHub integration - without it, you can't display
|
| 6 |
+
results in the GitHub Security tab.
|
| 7 |
+
"""
|
| 8 |
+
|
| 9 |
+
import json
|
| 10 |
+
import logging
|
| 11 |
+
from datetime import datetime, timezone
|
| 12 |
+
from typing import Any, Dict, List, Optional
|
| 13 |
+
|
| 14 |
+
logger = logging.getLogger("shield_agents.sarif")
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
class SARIFGenerator:
|
| 18 |
+
"""Generate SARIF 2.1.0 output from Shield Agents findings.
|
| 19 |
+
|
| 20 |
+
SARIF is the standard format for static analysis results exchange.
|
| 21 |
+
GitHub Security tab natively consumes SARIF files uploaded via
|
| 22 |
+
the code-scanning API.
|
| 23 |
+
"""
|
| 24 |
+
|
| 25 |
+
SARIF_SCHEMA = "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json"
|
| 26 |
+
SARIF_VERSION = "2.1.0"
|
| 27 |
+
|
| 28 |
+
# Map Shield Agents severity to SARIF level
|
| 29 |
+
SEVERITY_MAP = {
|
| 30 |
+
"CRITICAL": "error",
|
| 31 |
+
"HIGH": "error",
|
| 32 |
+
"MEDIUM": "warning",
|
| 33 |
+
"LOW": "note",
|
| 34 |
+
"INFO": "note",
|
| 35 |
+
}
|
| 36 |
+
|
| 37 |
+
def __init__(self, config=None):
|
| 38 |
+
self.config = config
|
| 39 |
+
|
| 40 |
+
def generate(self, findings: List[Dict[str, Any]], target_path: str = "") -> Dict[str, Any]:
|
| 41 |
+
"""Generate a SARIF report from findings.
|
| 42 |
+
|
| 43 |
+
Args:
|
| 44 |
+
findings: List of finding dictionaries.
|
| 45 |
+
target_path: Root path of the scanned project.
|
| 46 |
+
|
| 47 |
+
Returns:
|
| 48 |
+
SARIF 2.1.0 compliant dictionary.
|
| 49 |
+
"""
|
| 50 |
+
# Build rules from unique rule IDs
|
| 51 |
+
rules, rule_indices = self._build_rules(findings)
|
| 52 |
+
|
| 53 |
+
# Build results
|
| 54 |
+
results = []
|
| 55 |
+
for finding in findings:
|
| 56 |
+
result = self._build_result(finding, rule_indices, target_path)
|
| 57 |
+
results.append(result)
|
| 58 |
+
|
| 59 |
+
sarif = {
|
| 60 |
+
"$schema": self.SARIF_SCHEMA,
|
| 61 |
+
"version": self.SARIF_VERSION,
|
| 62 |
+
"runs": [
|
| 63 |
+
{
|
| 64 |
+
"tool": {
|
| 65 |
+
"driver": {
|
| 66 |
+
"name": "Shield Agents",
|
| 67 |
+
"version": "2.0.0",
|
| 68 |
+
"semanticVersion": "2.0.0",
|
| 69 |
+
"informationUri": "https://github.com/shield-agents/shield-agents",
|
| 70 |
+
"rules": rules,
|
| 71 |
+
"organization": "Shield Agents",
|
| 72 |
+
}
|
| 73 |
+
},
|
| 74 |
+
"results": results,
|
| 75 |
+
"invocations": [
|
| 76 |
+
{
|
| 77 |
+
"executionSuccessful": True,
|
| 78 |
+
"startTimeUtc": datetime.now(timezone.utc).isoformat(),
|
| 79 |
+
"endTimeUtc": datetime.now(timezone.utc).isoformat(),
|
| 80 |
+
}
|
| 81 |
+
],
|
| 82 |
+
}
|
| 83 |
+
],
|
| 84 |
+
}
|
| 85 |
+
|
| 86 |
+
logger.info(f"Generated SARIF report with {len(results)} results and {len(rules)} rules")
|
| 87 |
+
return sarif
|
| 88 |
+
|
| 89 |
+
def generate_string(self, findings: List[Dict[str, Any]], target_path: str = "") -> str:
|
| 90 |
+
"""Generate SARIF report as JSON string.
|
| 91 |
+
|
| 92 |
+
Args:
|
| 93 |
+
findings: List of finding dictionaries.
|
| 94 |
+
target_path: Root path of the scanned project.
|
| 95 |
+
|
| 96 |
+
Returns:
|
| 97 |
+
SARIF JSON string.
|
| 98 |
+
"""
|
| 99 |
+
return json.dumps(self.generate(findings, target_path), indent=2)
|
| 100 |
+
|
| 101 |
+
def save(self, findings: List[Dict[str, Any]], output_path: str, target_path: str = "") -> str:
|
| 102 |
+
"""Generate and save SARIF report to file.
|
| 103 |
+
|
| 104 |
+
Args:
|
| 105 |
+
findings: List of finding dictionaries.
|
| 106 |
+
output_path: Path to save the SARIF file.
|
| 107 |
+
target_path: Root path of the scanned project.
|
| 108 |
+
|
| 109 |
+
Returns:
|
| 110 |
+
Path to the saved file.
|
| 111 |
+
"""
|
| 112 |
+
sarif = self.generate(findings, target_path)
|
| 113 |
+
|
| 114 |
+
with open(output_path, "w") as f:
|
| 115 |
+
json.dump(sarif, f, indent=2)
|
| 116 |
+
|
| 117 |
+
logger.info(f"SARIF report saved to {output_path}")
|
| 118 |
+
return output_path
|
| 119 |
+
|
| 120 |
+
def _build_rules(self, findings: List[Dict[str, Any]]) -> tuple:
|
| 121 |
+
"""Build SARIF rules from findings.
|
| 122 |
+
|
| 123 |
+
Args:
|
| 124 |
+
findings: List of finding dictionaries.
|
| 125 |
+
|
| 126 |
+
Returns:
|
| 127 |
+
Tuple of (rules_list, rule_index_map).
|
| 128 |
+
"""
|
| 129 |
+
seen_rules = {}
|
| 130 |
+
rules = []
|
| 131 |
+
rule_indices = {}
|
| 132 |
+
|
| 133 |
+
for finding in findings:
|
| 134 |
+
rule_id = finding.get("rule_id", finding.get("id", "unknown"))
|
| 135 |
+
category = finding.get("category", "unknown")
|
| 136 |
+
|
| 137 |
+
# Create a composite rule key
|
| 138 |
+
rule_key = f"{rule_id}:{category}"
|
| 139 |
+
|
| 140 |
+
if rule_key not in seen_rules:
|
| 141 |
+
rule_index = len(rules)
|
| 142 |
+
seen_rules[rule_key] = rule_index
|
| 143 |
+
rule_indices[rule_key] = rule_index
|
| 144 |
+
|
| 145 |
+
rule = {
|
| 146 |
+
"id": rule_id,
|
| 147 |
+
"name": finding.get("title", "Unknown"),
|
| 148 |
+
"shortDescription": {
|
| 149 |
+
"text": finding.get("title", "Unknown finding"),
|
| 150 |
+
},
|
| 151 |
+
"fullDescription": {
|
| 152 |
+
"text": finding.get("description", finding.get("title", "No description")),
|
| 153 |
+
},
|
| 154 |
+
"helpUri": f"https://cwe.mitre.org/data/definitions/{finding.get('cwe', '0').replace('CWE-', '')}.html" if finding.get("cwe") else "",
|
| 155 |
+
"properties": {
|
| 156 |
+
"category": category,
|
| 157 |
+
"severity": finding.get("severity", "MEDIUM"),
|
| 158 |
+
},
|
| 159 |
+
}
|
| 160 |
+
|
| 161 |
+
# Add remediation as help text
|
| 162 |
+
if finding.get("remediation"):
|
| 163 |
+
rule["help"] = {
|
| 164 |
+
"text": finding["remediation"],
|
| 165 |
+
}
|
| 166 |
+
|
| 167 |
+
rules.append(rule)
|
| 168 |
+
|
| 169 |
+
return rules, rule_indices
|
| 170 |
+
|
| 171 |
+
def _build_result(self, finding: Dict[str, Any], rule_indices: Dict[str, int], target_path: str = "") -> Dict[str, Any]:
|
| 172 |
+
"""Build a SARIF result entry from a finding.
|
| 173 |
+
|
| 174 |
+
Args:
|
| 175 |
+
finding: Finding dictionary.
|
| 176 |
+
rule_indices: Map of rule keys to indices.
|
| 177 |
+
target_path: Root path of the scanned project.
|
| 178 |
+
|
| 179 |
+
Returns:
|
| 180 |
+
SARIF result dictionary.
|
| 181 |
+
"""
|
| 182 |
+
rule_id = finding.get("rule_id", finding.get("id", "unknown"))
|
| 183 |
+
category = finding.get("category", "unknown")
|
| 184 |
+
rule_key = f"{rule_id}:{category}"
|
| 185 |
+
|
| 186 |
+
result = {
|
| 187 |
+
"ruleId": rule_id,
|
| 188 |
+
"ruleIndex": rule_indices.get(rule_key, 0),
|
| 189 |
+
"level": self.SEVERITY_MAP.get(finding.get("severity", "MEDIUM").upper(), "warning"),
|
| 190 |
+
"message": {
|
| 191 |
+
"text": finding.get("description", finding.get("title", "No description")),
|
| 192 |
+
},
|
| 193 |
+
}
|
| 194 |
+
|
| 195 |
+
# Add location if file info is available
|
| 196 |
+
file_path = finding.get("file", "")
|
| 197 |
+
if file_path:
|
| 198 |
+
# Make path relative to target
|
| 199 |
+
if target_path and file_path.startswith(target_path):
|
| 200 |
+
file_path = file_path[len(target_path):].lstrip("/")
|
| 201 |
+
|
| 202 |
+
location = {
|
| 203 |
+
"physicalLocation": {
|
| 204 |
+
"artifactLocation": {
|
| 205 |
+
"uri": file_path,
|
| 206 |
+
},
|
| 207 |
+
},
|
| 208 |
+
}
|
| 209 |
+
|
| 210 |
+
# Add line number if available
|
| 211 |
+
line = finding.get("line")
|
| 212 |
+
if isinstance(line, int) and line > 0:
|
| 213 |
+
location["physicalLocation"]["region"] = {
|
| 214 |
+
"startLine": line,
|
| 215 |
+
}
|
| 216 |
+
|
| 217 |
+
result["locations"] = [location]
|
| 218 |
+
|
| 219 |
+
# Add code snippet as context region
|
| 220 |
+
if finding.get("code_snippet"):
|
| 221 |
+
if "locations" not in result:
|
| 222 |
+
result["locations"] = [{}]
|
| 223 |
+
if "physicalLocation" not in result["locations"][0]:
|
| 224 |
+
result["locations"][0]["physicalLocation"] = {}
|
| 225 |
+
result["locations"][0]["physicalLocation"]["contextRegion"] = {
|
| 226 |
+
"snippet": {
|
| 227 |
+
"text": finding["code_snippet"],
|
| 228 |
+
},
|
| 229 |
+
}
|
| 230 |
+
|
| 231 |
+
# Add additional properties
|
| 232 |
+
result["properties"] = {}
|
| 233 |
+
for key in ["agent", "source", "confidence", "cwe", "owasp", "remediation"]:
|
| 234 |
+
if key in finding:
|
| 235 |
+
result["properties"][key] = finding[key]
|
| 236 |
+
|
| 237 |
+
if finding.get("sources"):
|
| 238 |
+
result["properties"]["sources"] = finding["sources"]
|
| 239 |
+
|
| 240 |
+
return result
|
shield_agents/scanners/__init__.py
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Security scanners for Shield Agents."""
|
| 2 |
+
|
| 3 |
+
from .sast import SASTScanner
|
| 4 |
+
from .secrets import SecretsScanner
|
| 5 |
+
|
| 6 |
+
__all__ = ["SASTScanner", "SecretsScanner"]
|
shield_agents/scanners/sast.py
ADDED
|
@@ -0,0 +1,319 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Static Application Security Testing (SAST) Scanner for Shield Agents.
|
| 3 |
+
|
| 4 |
+
Implements 10 detection rules covering the most common vulnerability categories.
|
| 5 |
+
Returns structured findings compatible with the deduplication engine.
|
| 6 |
+
"""
|
| 7 |
+
|
| 8 |
+
import re
|
| 9 |
+
import logging
|
| 10 |
+
from dataclasses import dataclass
|
| 11 |
+
from pathlib import Path
|
| 12 |
+
from typing import Any, Dict, List, Optional, Tuple
|
| 13 |
+
|
| 14 |
+
logger = logging.getLogger("shield_agents.sast")
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
@dataclass
|
| 18 |
+
class SASTRule:
|
| 19 |
+
"""A SAST detection rule."""
|
| 20 |
+
id: str
|
| 21 |
+
name: str
|
| 22 |
+
description: str
|
| 23 |
+
severity: str
|
| 24 |
+
category: str
|
| 25 |
+
cwe: str
|
| 26 |
+
patterns: List[str]
|
| 27 |
+
remediation: str
|
| 28 |
+
languages: List[str] = None
|
| 29 |
+
|
| 30 |
+
def __post_init__(self):
|
| 31 |
+
if self.languages is None:
|
| 32 |
+
self.languages = ["*"]
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
# The 10 SAST detection rules
|
| 36 |
+
SAST_RULES: List[SASTRule] = [
|
| 37 |
+
SASTRule(
|
| 38 |
+
id="SAST-001",
|
| 39 |
+
name="SQL Injection",
|
| 40 |
+
description="SQL query constructed with string formatting or concatenation, allowing injection of arbitrary SQL",
|
| 41 |
+
severity="CRITICAL",
|
| 42 |
+
category="injection",
|
| 43 |
+
cwe="CWE-89",
|
| 44 |
+
patterns=[
|
| 45 |
+
r'cursor\.execute\s*\(\s*[f"\'].*(?:\+|%s|\.format|f["\'])',
|
| 46 |
+
r'\.raw\s*\(\s*[f"\'].*(?:\+|%s|\.format|f["\'])',
|
| 47 |
+
r'execute\s*\(\s*[f"\']SELECT.*\+\s*[\w]+',
|
| 48 |
+
r'execute\s*\(\s*[f"\']INSERT.*\+\s*[\w]+',
|
| 49 |
+
r'execute\s*\(\s*[f"\']UPDATE.*\+\s*[\w]+',
|
| 50 |
+
r'execute\s*\(\s*[f"\']DELETE.*\+\s*[\w]+',
|
| 51 |
+
r'f["\'].*SELECT.*FROM.*WHERE.*\{',
|
| 52 |
+
r'f["\'].*INSERT.*INTO.*VALUES.*\{',
|
| 53 |
+
],
|
| 54 |
+
remediation="Use parameterized queries with placeholders: cursor.execute('SELECT * FROM users WHERE id = %s', (user_id,))",
|
| 55 |
+
languages=["python"],
|
| 56 |
+
),
|
| 57 |
+
SASTRule(
|
| 58 |
+
id="SAST-002",
|
| 59 |
+
name="Cross-Site Scripting (XSS)",
|
| 60 |
+
description="User input reflected in HTML without proper escaping",
|
| 61 |
+
severity="HIGH",
|
| 62 |
+
category="xss",
|
| 63 |
+
cwe="CWE-79",
|
| 64 |
+
patterns=[
|
| 65 |
+
r'innerHTML\s*=',
|
| 66 |
+
r'document\.write\s*\(',
|
| 67 |
+
r'render_template_string\s*\(',
|
| 68 |
+
r'Markup\s*\(',
|
| 69 |
+
r'\|\s*safe\b',
|
| 70 |
+
r'v-html\s*=',
|
| 71 |
+
],
|
| 72 |
+
remediation="Sanitize user input before rendering. Use textContent instead of innerHTML. Apply output encoding.",
|
| 73 |
+
languages=["javascript", "python", "html"],
|
| 74 |
+
),
|
| 75 |
+
SASTRule(
|
| 76 |
+
id="SAST-003",
|
| 77 |
+
name="Command Injection",
|
| 78 |
+
description="OS command execution with potentially user-controlled input",
|
| 79 |
+
severity="CRITICAL",
|
| 80 |
+
category="injection",
|
| 81 |
+
cwe="CWE-78",
|
| 82 |
+
patterns=[
|
| 83 |
+
r'os\.system\s*\(',
|
| 84 |
+
r'os\.popen\s*\(',
|
| 85 |
+
r'subprocess\.(call|run|Popen|check_output)\s*\([^)]*shell\s*=\s*True',
|
| 86 |
+
r'subprocess\.(call|run|Popen)\s*\(\s*["\']',
|
| 87 |
+
r'exec\s*\(',
|
| 88 |
+
],
|
| 89 |
+
remediation="Use subprocess with shell=False and list arguments. Never pass user input to shell commands directly.",
|
| 90 |
+
languages=["python"],
|
| 91 |
+
),
|
| 92 |
+
SASTRule(
|
| 93 |
+
id="SAST-004",
|
| 94 |
+
name="Path Traversal",
|
| 95 |
+
description="File path constructed with user input, allowing directory traversal",
|
| 96 |
+
severity="HIGH",
|
| 97 |
+
category="path-traversal",
|
| 98 |
+
cwe="CWE-22",
|
| 99 |
+
patterns=[
|
| 100 |
+
r'open\s*\(\s*["\'].*\+\s*\w+',
|
| 101 |
+
r'open\s*\(\s*.*\+\s*request\.',
|
| 102 |
+
r'open\s*\(\s*.*\.format\s*\(.*request\.',
|
| 103 |
+
r'os\.path\.join\s*\([^)]*request\.',
|
| 104 |
+
r'send_file\s*\(\s*request\.',
|
| 105 |
+
r'send_file\s*\([^)]*request\.',
|
| 106 |
+
r'f["\'].*\/.*\{.*request',
|
| 107 |
+
r'["\']/var/[a-z]+/["\']\s*\+\s*\w+',
|
| 108 |
+
],
|
| 109 |
+
remediation="Validate and sanitize file paths. Use allowlists for permitted directories. Avoid constructing paths from user input.",
|
| 110 |
+
languages=["python", "javascript"],
|
| 111 |
+
),
|
| 112 |
+
SASTRule(
|
| 113 |
+
id="SAST-005",
|
| 114 |
+
name="Insecure Deserialization",
|
| 115 |
+
description="Deserialization of untrusted data can lead to remote code execution",
|
| 116 |
+
severity="CRITICAL",
|
| 117 |
+
category="deserialization",
|
| 118 |
+
cwe="CWE-502",
|
| 119 |
+
patterns=[
|
| 120 |
+
r'pickle\.loads?\s*\(',
|
| 121 |
+
r'yaml\.load\s*\([^)]*\)(?!.*Loader)',
|
| 122 |
+
r'marshal\.loads?\s*\(',
|
| 123 |
+
r'shelve\.open\s*\(',
|
| 124 |
+
],
|
| 125 |
+
remediation="Use safe serialization formats like JSON. For YAML, use yaml.safe_load(). Never deserialize untrusted data with pickle.",
|
| 126 |
+
languages=["python"],
|
| 127 |
+
),
|
| 128 |
+
SASTRule(
|
| 129 |
+
id="SAST-006",
|
| 130 |
+
name="Weak Cryptography",
|
| 131 |
+
description="Use of weak or broken cryptographic algorithms",
|
| 132 |
+
severity="MEDIUM",
|
| 133 |
+
category="cryptography",
|
| 134 |
+
cwe="CWE-327",
|
| 135 |
+
patterns=[
|
| 136 |
+
r'hashlib\.(md5|sha1)\s*\(',
|
| 137 |
+
r'DES|RC4|Blowfish',
|
| 138 |
+
r'MODE_ECB',
|
| 139 |
+
r'random\.(random|randint|choice|shuffle)\s*\((?!.*#.*security)',
|
| 140 |
+
r'cryptography\.hazmat.*MD5',
|
| 141 |
+
],
|
| 142 |
+
remediation="Use SHA-256 or stronger for hashing. Use AES-GCM or ChaCha20 for encryption. Use secrets module for random values.",
|
| 143 |
+
languages=["python"],
|
| 144 |
+
),
|
| 145 |
+
SASTRule(
|
| 146 |
+
id="SAST-007",
|
| 147 |
+
name="Authentication Issues",
|
| 148 |
+
description="Weak or improper authentication implementation",
|
| 149 |
+
severity="HIGH",
|
| 150 |
+
category="authentication",
|
| 151 |
+
cwe="CWE-287",
|
| 152 |
+
patterns=[
|
| 153 |
+
r'verify_password\s*=\s*False',
|
| 154 |
+
r'check_password\s*=\s*True',
|
| 155 |
+
r'assert\s+.*authenticated',
|
| 156 |
+
r'@login_required',
|
| 157 |
+
r'session\[.*\]\s*=\s*True(?!.*verify)',
|
| 158 |
+
r'jwt\.decode\s*\([^)]*\)(?!.*algorithms)',
|
| 159 |
+
],
|
| 160 |
+
remediation="Implement proper authentication checks. Use established authentication libraries. Never bypass auth with assertions.",
|
| 161 |
+
languages=["python"],
|
| 162 |
+
),
|
| 163 |
+
SASTRule(
|
| 164 |
+
id="SAST-008",
|
| 165 |
+
name="Hardcoded Credentials",
|
| 166 |
+
description="Credentials hardcoded in source code",
|
| 167 |
+
severity="HIGH",
|
| 168 |
+
category="credentials",
|
| 169 |
+
cwe="CWE-798",
|
| 170 |
+
patterns=[
|
| 171 |
+
r'(?:password|passwd|pwd)\s*=\s*["\'][^"\']{4,}["\']',
|
| 172 |
+
r'(?:api_key|apikey|api_secret)\s*=\s*["\'][^"\']{8,}["\']',
|
| 173 |
+
r'(?:secret_key|secretkey|secret)\s*=\s*["\'][^"\']{8,}["\']',
|
| 174 |
+
r'(?:token|auth_token|access_token)\s*=\s*["\'][^"\']{8,}["\']',
|
| 175 |
+
r'(?:database_url|db_password)\s*=\s*["\'][^"\']{8,}["\']',
|
| 176 |
+
],
|
| 177 |
+
remediation="Move credentials to environment variables or a secure vault. Use os.environ.get() or a secrets manager.",
|
| 178 |
+
languages=["python", "javascript"],
|
| 179 |
+
),
|
| 180 |
+
SASTRule(
|
| 181 |
+
id="SAST-009",
|
| 182 |
+
name="Insecure SSL/TLS Configuration",
|
| 183 |
+
description="SSL/TLS certificate verification disabled",
|
| 184 |
+
severity="HIGH",
|
| 185 |
+
category="security-misconfiguration",
|
| 186 |
+
cwe="CWE-295",
|
| 187 |
+
patterns=[
|
| 188 |
+
r'verify\s*=\s*False',
|
| 189 |
+
r'ssl\._create_unverified_context',
|
| 190 |
+
r'CERT_NONE',
|
| 191 |
+
r'requests\.get\s*\([^)]*verify\s*=\s*False',
|
| 192 |
+
r'disable.*ssl|disable.*tls|skip.*verification',
|
| 193 |
+
],
|
| 194 |
+
remediation="Always verify SSL certificates. Never use verify=False in production. Use proper certificate bundles.",
|
| 195 |
+
languages=["python"],
|
| 196 |
+
),
|
| 197 |
+
SASTRule(
|
| 198 |
+
id="SAST-010",
|
| 199 |
+
name="Server-Side Request Forgery (SSRF)",
|
| 200 |
+
description="Application makes requests to user-specified URLs",
|
| 201 |
+
severity="HIGH",
|
| 202 |
+
category="ssrf",
|
| 203 |
+
cwe="CWE-918",
|
| 204 |
+
patterns=[
|
| 205 |
+
r'requests\.(get|post|put|delete)\s*\(\s*request\.',
|
| 206 |
+
r'requests\.(get|post|put|delete)\s*\([^)]*request\.args',
|
| 207 |
+
r'requests\.(get|post)\s*\(\s*url\s*\)',
|
| 208 |
+
r'urllib\.request\.urlopen\s*\(\s*request\.',
|
| 209 |
+
r'http\.client.*request\.',
|
| 210 |
+
r'fetch\s*\(\s*.*request\.',
|
| 211 |
+
r'urlopen\s*\(\s*.*\.format\s*\(.*request',
|
| 212 |
+
r'return\s+requests\.\w+\([^)]*\)\.text',
|
| 213 |
+
],
|
| 214 |
+
remediation="Validate URLs against an allowlist. Block requests to internal IPs. Use URL parsing to validate scheme and host.",
|
| 215 |
+
languages=["python", "javascript"],
|
| 216 |
+
),
|
| 217 |
+
]
|
| 218 |
+
|
| 219 |
+
|
| 220 |
+
class SASTScanner:
|
| 221 |
+
"""Static Application Security Testing scanner.
|
| 222 |
+
|
| 223 |
+
Scans source code files using pattern-based detection rules.
|
| 224 |
+
"""
|
| 225 |
+
|
| 226 |
+
def __init__(self, config=None):
|
| 227 |
+
self.config = config
|
| 228 |
+
self.rules = SAST_RULES
|
| 229 |
+
self.findings: List[Dict[str, Any]] = []
|
| 230 |
+
|
| 231 |
+
def scan_file(self, file_path: str, content: Optional[str] = None) -> List[Dict[str, Any]]:
|
| 232 |
+
"""Scan a single file for vulnerabilities.
|
| 233 |
+
|
| 234 |
+
Args:
|
| 235 |
+
file_path: Path to the file.
|
| 236 |
+
content: Optional file content (will be read if not provided).
|
| 237 |
+
|
| 238 |
+
Returns:
|
| 239 |
+
List of findings for this file.
|
| 240 |
+
"""
|
| 241 |
+
from ..utils.helpers import read_file_safe
|
| 242 |
+
|
| 243 |
+
if content is None:
|
| 244 |
+
content = read_file_safe(file_path)
|
| 245 |
+
if content is None:
|
| 246 |
+
return []
|
| 247 |
+
|
| 248 |
+
file_findings = []
|
| 249 |
+
lines = content.split("\n")
|
| 250 |
+
|
| 251 |
+
for rule in self.rules:
|
| 252 |
+
for pattern in rule.patterns:
|
| 253 |
+
try:
|
| 254 |
+
for match in re.finditer(pattern, content, re.IGNORECASE | re.DOTALL):
|
| 255 |
+
line_num = content[:match.start()].count("\n") + 1
|
| 256 |
+
line_content = lines[line_num - 1].strip() if line_num <= len(lines) else match.group(0)
|
| 257 |
+
|
| 258 |
+
# Skip very short matches that are likely false positives
|
| 259 |
+
if len(match.group(0)) < 3:
|
| 260 |
+
continue
|
| 261 |
+
|
| 262 |
+
finding = {
|
| 263 |
+
"id": f"{rule.id}-{len(file_findings) + 1}",
|
| 264 |
+
"rule_id": rule.id,
|
| 265 |
+
"title": rule.name,
|
| 266 |
+
"description": rule.description,
|
| 267 |
+
"severity": rule.severity,
|
| 268 |
+
"category": rule.category,
|
| 269 |
+
"cwe": rule.cwe,
|
| 270 |
+
"file": file_path,
|
| 271 |
+
"line": line_num,
|
| 272 |
+
"code_snippet": line_content[:200],
|
| 273 |
+
"remediation": rule.remediation,
|
| 274 |
+
"source": "SAST",
|
| 275 |
+
"agent": "SASTScanner",
|
| 276 |
+
"confidence": 0.85,
|
| 277 |
+
}
|
| 278 |
+
file_findings.append(finding)
|
| 279 |
+
except re.error:
|
| 280 |
+
continue
|
| 281 |
+
|
| 282 |
+
self.findings.extend(file_findings)
|
| 283 |
+
logger.info(f"SAST: Found {len(file_findings)} issues in {file_path}")
|
| 284 |
+
return file_findings
|
| 285 |
+
|
| 286 |
+
def scan_directory(self, directory: str, excluded_dirs: Optional[List[str]] = None) -> List[Dict[str, Any]]:
|
| 287 |
+
"""Scan all source files in a directory.
|
| 288 |
+
|
| 289 |
+
Args:
|
| 290 |
+
directory: Directory to scan.
|
| 291 |
+
excluded_dirs: Directory names to exclude.
|
| 292 |
+
|
| 293 |
+
Returns:
|
| 294 |
+
List of all findings.
|
| 295 |
+
"""
|
| 296 |
+
from ..utils.helpers import find_files, read_file_safe
|
| 297 |
+
|
| 298 |
+
if excluded_dirs is None:
|
| 299 |
+
excluded_dirs = ["node_modules", ".git", "__pycache__", ".venv", "venv", "dist", "build"]
|
| 300 |
+
|
| 301 |
+
source_extensions = [
|
| 302 |
+
".py", ".js", ".ts", ".jsx", ".tsx", ".java", ".go", ".rb",
|
| 303 |
+
".php", ".c", ".cpp", ".cs", ".rs", ".swift", ".kt",
|
| 304 |
+
".html", ".htm", ".xml", ".yaml", ".yml", ".json", ".sql",
|
| 305 |
+
]
|
| 306 |
+
|
| 307 |
+
files = find_files(directory, extensions=source_extensions, exclude_dirs=excluded_dirs)
|
| 308 |
+
|
| 309 |
+
all_findings = []
|
| 310 |
+
for file_path in files:
|
| 311 |
+
findings = self.scan_file(file_path)
|
| 312 |
+
all_findings.extend(findings)
|
| 313 |
+
|
| 314 |
+
logger.info(f"SAST: Scanned {len(files)} files, found {len(all_findings)} total issues")
|
| 315 |
+
return all_findings
|
| 316 |
+
|
| 317 |
+
def clear_findings(self):
|
| 318 |
+
"""Clear all findings."""
|
| 319 |
+
self.findings = []
|
shield_agents/scanners/secrets.py
ADDED
|
@@ -0,0 +1,324 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Secrets detection scanner for Shield Agents.
|
| 3 |
+
|
| 4 |
+
Detects hardcoded secrets, API keys, tokens, and other sensitive data
|
| 5 |
+
in source code. Supports 20+ secret types.
|
| 6 |
+
"""
|
| 7 |
+
|
| 8 |
+
import re
|
| 9 |
+
import logging
|
| 10 |
+
from typing import Any, Dict, List, Optional
|
| 11 |
+
|
| 12 |
+
logger = logging.getLogger("shield_agents.secrets")
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
# Secret detection patterns - 20+ types
|
| 16 |
+
SECRET_PATTERNS = [
|
| 17 |
+
# AWS
|
| 18 |
+
{
|
| 19 |
+
"name": "AWS Access Key ID",
|
| 20 |
+
"pattern": r'AWS_KEY_PATTERN_PLACEHOLDER',
|
| 21 |
+
"severity": "CRITICAL",
|
| 22 |
+
"category": "cloud-credentials",
|
| 23 |
+
},
|
| 24 |
+
{
|
| 25 |
+
"name": "AWS Secret Access Key",
|
| 26 |
+
"pattern": r'(?i)aws[_\-]?secret[_\-]?access[_\-]?key\s*[=:]\s*["\']?[A-Za-z0-9/+=]{40}["\']?',
|
| 27 |
+
"severity": "CRITICAL",
|
| 28 |
+
"category": "cloud-credentials",
|
| 29 |
+
},
|
| 30 |
+
# GitHub
|
| 31 |
+
{
|
| 32 |
+
"name": "GitHub Personal Access Token",
|
| 33 |
+
"pattern": r'GITHUB_TOKEN_PATTERN_PLACEHOLDER',
|
| 34 |
+
"severity": "CRITICAL",
|
| 35 |
+
"category": "vcs-credentials",
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"name": "GitHub OAuth Access Token",
|
| 39 |
+
"pattern": r'gho_[0-9a-zA-Z]{36}',
|
| 40 |
+
"severity": "CRITICAL",
|
| 41 |
+
"category": "vcs-credentials",
|
| 42 |
+
},
|
| 43 |
+
{
|
| 44 |
+
"name": "GitHub Fine-Grained PAT",
|
| 45 |
+
"pattern": r'github_pat_[0-9a-zA-Z_]{82}',
|
| 46 |
+
"severity": "CRITICAL",
|
| 47 |
+
"category": "vcs-credentials",
|
| 48 |
+
},
|
| 49 |
+
# Google
|
| 50 |
+
{
|
| 51 |
+
"name": "Google API Key",
|
| 52 |
+
"pattern": r'AIza[0-9A-Za-z\-_]{35}',
|
| 53 |
+
"severity": "HIGH",
|
| 54 |
+
"category": "cloud-credentials",
|
| 55 |
+
},
|
| 56 |
+
{
|
| 57 |
+
"name": "Google OAuth Access Token",
|
| 58 |
+
"pattern": r'ya29\.[0-9A-Za-z\-_]+',
|
| 59 |
+
"severity": "HIGH",
|
| 60 |
+
"category": "cloud-credentials",
|
| 61 |
+
},
|
| 62 |
+
# Slack
|
| 63 |
+
{
|
| 64 |
+
"name": "Slack Bot Token",
|
| 65 |
+
"pattern": r'xoxb-[0-9]{10,13}-[0-9]{10,13}-[0-9a-zA-Z]{24}',
|
| 66 |
+
"severity": "CRITICAL",
|
| 67 |
+
"category": "messaging-credentials",
|
| 68 |
+
},
|
| 69 |
+
{
|
| 70 |
+
"name": "Slack Webhook URL",
|
| 71 |
+
"pattern": r'https://hooks\.slack\.com/services/T[a-zA-Z0-9_]{8,}/B[a-zA-Z0-9_]{8,}/[a-zA-Z0-9_]{24}',
|
| 72 |
+
"severity": "HIGH",
|
| 73 |
+
"category": "messaging-credentials",
|
| 74 |
+
},
|
| 75 |
+
# Stripe
|
| 76 |
+
{
|
| 77 |
+
"name": "Stripe Secret Key",
|
| 78 |
+
"pattern": r'STRIPE_KEY_PATTERN_PLACEHOLDER',
|
| 79 |
+
"severity": "CRITICAL",
|
| 80 |
+
"category": "payment-credentials",
|
| 81 |
+
},
|
| 82 |
+
{
|
| 83 |
+
"name": "Stripe Publishable Key",
|
| 84 |
+
"pattern": r'pk_live_[0-9a-zA-Z]{24,}',
|
| 85 |
+
"severity": "MEDIUM",
|
| 86 |
+
"category": "payment-credentials",
|
| 87 |
+
},
|
| 88 |
+
# Database Connection Strings
|
| 89 |
+
{
|
| 90 |
+
"name": "Database Connection String",
|
| 91 |
+
"pattern": r'(?i)(?:mysql|postgres|mongodb|redis)://[^\s\'"]+:[^\s\'"]+@[^\s\'"]+',
|
| 92 |
+
"severity": "CRITICAL",
|
| 93 |
+
"category": "database-credentials",
|
| 94 |
+
},
|
| 95 |
+
{
|
| 96 |
+
"name": "MongoDB Connection String",
|
| 97 |
+
"pattern": r'mongodb(\+srv)?://[^\s\'"]+:[^\s\'"]+@[^\s\'"]+',
|
| 98 |
+
"severity": "CRITICAL",
|
| 99 |
+
"category": "database-credentials",
|
| 100 |
+
},
|
| 101 |
+
# JWT
|
| 102 |
+
{
|
| 103 |
+
"name": "JSON Web Token",
|
| 104 |
+
"pattern": r'eyJ[A-Za-z0-9\-_]+\.eyJ[A-Za-z0-9\-_]+\.[A-Za-z0-9\-_]+',
|
| 105 |
+
"severity": "MEDIUM",
|
| 106 |
+
"category": "auth-tokens",
|
| 107 |
+
},
|
| 108 |
+
# Private Keys
|
| 109 |
+
{
|
| 110 |
+
"name": "RSA Private Key",
|
| 111 |
+
"pattern": r'-----BEGIN RSA PRIVATE KEY-----',
|
| 112 |
+
"severity": "CRITICAL",
|
| 113 |
+
"category": "crypto-keys",
|
| 114 |
+
},
|
| 115 |
+
{
|
| 116 |
+
"name": "SSH Private Key",
|
| 117 |
+
"pattern": r'-----BEGIN OPENSSH PRIVATE KEY-----',
|
| 118 |
+
"severity": "CRITICAL",
|
| 119 |
+
"category": "crypto-keys",
|
| 120 |
+
},
|
| 121 |
+
{
|
| 122 |
+
"name": "EC Private Key",
|
| 123 |
+
"pattern": r'-----BEGIN EC PRIVATE KEY-----',
|
| 124 |
+
"severity": "CRITICAL",
|
| 125 |
+
"category": "crypto-keys",
|
| 126 |
+
},
|
| 127 |
+
{
|
| 128 |
+
"name": "PGP Private Key Block",
|
| 129 |
+
"pattern": r'-----BEGIN PGP PRIVATE KEY BLOCK-----',
|
| 130 |
+
"severity": "CRITICAL",
|
| 131 |
+
"category": "crypto-keys",
|
| 132 |
+
},
|
| 133 |
+
# Heroku
|
| 134 |
+
{
|
| 135 |
+
"name": "Heroku API Key",
|
| 136 |
+
"pattern": r'(?i)heroku[_\-]?api[_\-]?key\s*[=:]\s*["\']?[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}["\']?',
|
| 137 |
+
"severity": "HIGH",
|
| 138 |
+
"category": "cloud-credentials",
|
| 139 |
+
},
|
| 140 |
+
# Twilio
|
| 141 |
+
{
|
| 142 |
+
"name": "Twilio Account SID",
|
| 143 |
+
"pattern": r'AC[a-z0-9]{32}',
|
| 144 |
+
"severity": "HIGH",
|
| 145 |
+
"category": "messaging-credentials",
|
| 146 |
+
},
|
| 147 |
+
{
|
| 148 |
+
"name": "Twilio Auth Token",
|
| 149 |
+
"pattern": r'(?i)twilio[_\-]?auth[_\-]?token\s*[=:]\s*["\']?[0-9a-fA-F]{32}["\']?',
|
| 150 |
+
"severity": "CRITICAL",
|
| 151 |
+
"category": "messaging-credentials",
|
| 152 |
+
},
|
| 153 |
+
# SendGrid
|
| 154 |
+
{
|
| 155 |
+
"name": "SendGrid API Key",
|
| 156 |
+
"pattern": r'SG\.[0-9a-zA-Z\-_]{22}\.[0-9a-zA-Z\-_]{43}',
|
| 157 |
+
"severity": "CRITICAL",
|
| 158 |
+
"category": "messaging-credentials",
|
| 159 |
+
},
|
| 160 |
+
# Generic
|
| 161 |
+
{
|
| 162 |
+
"name": "Generic Secret Assignment",
|
| 163 |
+
"pattern": r'(?i)(?:password|passwd|pwd|secret|token|api_key|apikey|auth_key|access_key|private_key)\s*[=:]\s*["\'][^"\']{8,}["\']',
|
| 164 |
+
"severity": "HIGH",
|
| 165 |
+
"category": "generic-secrets",
|
| 166 |
+
},
|
| 167 |
+
{
|
| 168 |
+
"name": "Authorization Header Bearer Token",
|
| 169 |
+
"pattern": r'(?i)authorization\s*[:=]\s*["\']?Bearer\s+[A-Za-z0-9\-_.~+/]+=*["\']?',
|
| 170 |
+
"severity": "HIGH",
|
| 171 |
+
"category": "auth-tokens",
|
| 172 |
+
},
|
| 173 |
+
]
|
| 174 |
+
|
| 175 |
+
# Entropy-based detection threshold
|
| 176 |
+
HIGH_ENTROPY_THRESHOLD = 4.5
|
| 177 |
+
|
| 178 |
+
|
| 179 |
+
class SecretsScanner:
|
| 180 |
+
"""Detects hardcoded secrets and sensitive data in source code."""
|
| 181 |
+
|
| 182 |
+
def __init__(self, config=None):
|
| 183 |
+
self.config = config
|
| 184 |
+
self.patterns = SECRET_PATTERNS
|
| 185 |
+
self.findings: List[Dict[str, Any]] = []
|
| 186 |
+
|
| 187 |
+
def _calculate_entropy(self, string: str) -> float:
|
| 188 |
+
"""Calculate Shannon entropy of a string.
|
| 189 |
+
|
| 190 |
+
Args:
|
| 191 |
+
string: Input string.
|
| 192 |
+
|
| 193 |
+
Returns:
|
| 194 |
+
Entropy value (higher = more random).
|
| 195 |
+
"""
|
| 196 |
+
import math
|
| 197 |
+
from collections import Counter
|
| 198 |
+
|
| 199 |
+
if not string:
|
| 200 |
+
return 0.0
|
| 201 |
+
|
| 202 |
+
counts = Counter(string)
|
| 203 |
+
length = len(string)
|
| 204 |
+
entropy = 0.0
|
| 205 |
+
|
| 206 |
+
for count in counts.values():
|
| 207 |
+
probability = count / length
|
| 208 |
+
if probability > 0:
|
| 209 |
+
entropy -= probability * math.log2(probability)
|
| 210 |
+
|
| 211 |
+
return entropy
|
| 212 |
+
|
| 213 |
+
def scan_file(self, file_path: str, content: Optional[str] = None) -> List[Dict[str, Any]]:
|
| 214 |
+
"""Scan a single file for secrets.
|
| 215 |
+
|
| 216 |
+
Args:
|
| 217 |
+
file_path: Path to the file.
|
| 218 |
+
content: Optional file content (will be read if not provided).
|
| 219 |
+
|
| 220 |
+
Returns:
|
| 221 |
+
List of secret findings.
|
| 222 |
+
"""
|
| 223 |
+
from ..utils.helpers import read_file_safe
|
| 224 |
+
|
| 225 |
+
if content is None:
|
| 226 |
+
content = read_file_safe(file_path)
|
| 227 |
+
if content is None:
|
| 228 |
+
return []
|
| 229 |
+
|
| 230 |
+
file_findings = []
|
| 231 |
+
lines = content.split("\n")
|
| 232 |
+
|
| 233 |
+
for pattern_info in self.patterns:
|
| 234 |
+
try:
|
| 235 |
+
for match in re.finditer(pattern_info["pattern"], content, re.IGNORECASE):
|
| 236 |
+
line_num = content[:match.start()].count("\n") + 1
|
| 237 |
+
line_content = lines[line_num - 1].strip() if line_num <= len(lines) else ""
|
| 238 |
+
|
| 239 |
+
# Check entropy for generic patterns to reduce false positives
|
| 240 |
+
matched_text = match.group(0)
|
| 241 |
+
if pattern_info["category"] == "generic-secrets":
|
| 242 |
+
entropy = self._calculate_entropy(matched_text)
|
| 243 |
+
if entropy < HIGH_ENTROPY_THRESHOLD:
|
| 244 |
+
continue
|
| 245 |
+
|
| 246 |
+
# Mask the secret in the snippet
|
| 247 |
+
masked_snippet = self._mask_secret(line_content)
|
| 248 |
+
|
| 249 |
+
finding = {
|
| 250 |
+
"id": f"SEC-{len(file_findings) + 1}",
|
| 251 |
+
"title": pattern_info["name"],
|
| 252 |
+
"description": f"Potential {pattern_info['name']} detected",
|
| 253 |
+
"severity": pattern_info["severity"],
|
| 254 |
+
"category": pattern_info["category"],
|
| 255 |
+
"file": file_path,
|
| 256 |
+
"line": line_num,
|
| 257 |
+
"code_snippet": masked_snippet[:200],
|
| 258 |
+
"remediation": "Move this secret to an environment variable or secrets manager. Never commit secrets to version control.",
|
| 259 |
+
"source": "SecretsScanner",
|
| 260 |
+
"agent": "SecretsScanner",
|
| 261 |
+
"confidence": 0.9,
|
| 262 |
+
}
|
| 263 |
+
file_findings.append(finding)
|
| 264 |
+
except re.error:
|
| 265 |
+
continue
|
| 266 |
+
|
| 267 |
+
self.findings.extend(file_findings)
|
| 268 |
+
logger.info(f"Secrets: Found {len(file_findings)} secrets in {file_path}")
|
| 269 |
+
return file_findings
|
| 270 |
+
|
| 271 |
+
def scan_directory(self, directory: str, excluded_dirs: Optional[List[str]] = None) -> List[Dict[str, Any]]:
|
| 272 |
+
"""Scan all files in a directory for secrets.
|
| 273 |
+
|
| 274 |
+
Args:
|
| 275 |
+
directory: Directory to scan.
|
| 276 |
+
excluded_dirs: Directory names to exclude.
|
| 277 |
+
|
| 278 |
+
Returns:
|
| 279 |
+
List of all secret findings.
|
| 280 |
+
"""
|
| 281 |
+
from ..utils.helpers import find_files
|
| 282 |
+
|
| 283 |
+
if excluded_dirs is None:
|
| 284 |
+
excluded_dirs = ["node_modules", ".git", "__pycache__", ".venv", "venv", "dist", "build"]
|
| 285 |
+
|
| 286 |
+
# Scan all text file types
|
| 287 |
+
extensions = [
|
| 288 |
+
".py", ".js", ".ts", ".jsx", ".tsx", ".java", ".go", ".rb",
|
| 289 |
+
".php", ".c", ".cpp", ".cs", ".rs", ".swift", ".kt",
|
| 290 |
+
".html", ".css", ".json", ".yaml", ".yml", ".xml", ".ini",
|
| 291 |
+
".env", ".cfg", ".conf", ".toml", ".properties", ".sh", ".bash",
|
| 292 |
+
]
|
| 293 |
+
|
| 294 |
+
files = find_files(directory, extensions=extensions, exclude_dirs=excluded_dirs)
|
| 295 |
+
|
| 296 |
+
all_findings = []
|
| 297 |
+
for file_path in files:
|
| 298 |
+
findings = self.scan_file(file_path)
|
| 299 |
+
all_findings.extend(findings)
|
| 300 |
+
|
| 301 |
+
logger.info(f"Secrets: Scanned {len(files)} files, found {len(all_findings)} total secrets")
|
| 302 |
+
return all_findings
|
| 303 |
+
|
| 304 |
+
@staticmethod
|
| 305 |
+
def _mask_secret(text: str) -> str:
|
| 306 |
+
"""Mask potential secrets in text for safe display.
|
| 307 |
+
|
| 308 |
+
Args:
|
| 309 |
+
text: Text containing a potential secret.
|
| 310 |
+
|
| 311 |
+
Returns:
|
| 312 |
+
Text with secret value masked.
|
| 313 |
+
"""
|
| 314 |
+
# Mask values after = or : for common secret patterns
|
| 315 |
+
masked = re.sub(
|
| 316 |
+
r'(["\']?)([A-Za-z0-9/+=]{8,})(["\']?)',
|
| 317 |
+
r'\1***MASKED***\3',
|
| 318 |
+
text,
|
| 319 |
+
)
|
| 320 |
+
return masked
|
| 321 |
+
|
| 322 |
+
def clear_findings(self):
|
| 323 |
+
"""Clear all findings."""
|
| 324 |
+
self.findings = []
|
shield_agents/shieldignore.py
ADDED
|
@@ -0,0 +1,298 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
.shieldignore file support for Shield Agents.
|
| 3 |
+
|
| 4 |
+
Like .gitignore but for suppressing false positives.
|
| 5 |
+
Users can specify patterns to exclude specific findings from reports.
|
| 6 |
+
Without this, users get annoyed quickly by false positives.
|
| 7 |
+
"""
|
| 8 |
+
|
| 9 |
+
import logging
|
| 10 |
+
import re
|
| 11 |
+
from pathlib import Path
|
| 12 |
+
from typing import Any, Dict, List, Optional, Set, Tuple
|
| 13 |
+
|
| 14 |
+
logger = logging.getLogger("shield_agents.shieldignore")
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
class ShieldIgnore:
|
| 18 |
+
"""Parse and apply .shieldignore rules to filter findings.
|
| 19 |
+
|
| 20 |
+
The .shieldignore file supports these rule types:
|
| 21 |
+
- file:path/to/file.py # Ignore all findings in a specific file
|
| 22 |
+
- rule:SAST-001 # Ignore a specific rule
|
| 23 |
+
- category:sql_injection # Ignore all findings of a category
|
| 24 |
+
- severity:LOW # Ignore all findings below a severity
|
| 25 |
+
- id:VulnAgent-3 # Ignore a specific finding by ID
|
| 26 |
+
- path:*.test.* # Ignore findings in files matching pattern
|
| 27 |
+
- line:42:path/to/file.py # Ignore finding at specific line in file
|
| 28 |
+
- title:Assertion* # Ignore findings with matching title (glob)
|
| 29 |
+
"""
|
| 30 |
+
|
| 31 |
+
def __init__(self, target_path: str = "."):
|
| 32 |
+
"""Initialize ShieldIgnore.
|
| 33 |
+
|
| 34 |
+
Args:
|
| 35 |
+
target_path: Root path to search for .shieldignore file.
|
| 36 |
+
"""
|
| 37 |
+
self.target_path = Path(target_path)
|
| 38 |
+
self.rules: List[Dict[str, Any]] = []
|
| 39 |
+
self._loaded = False
|
| 40 |
+
self._stats = {
|
| 41 |
+
"rules_loaded": 0,
|
| 42 |
+
"findings_filtered": 0,
|
| 43 |
+
}
|
| 44 |
+
|
| 45 |
+
def load(self, ignore_path: Optional[str] = None) -> None:
|
| 46 |
+
"""Load .shieldignore rules from file.
|
| 47 |
+
|
| 48 |
+
Args:
|
| 49 |
+
ignore_path: Path to the ignore file. Defaults to .shieldignore in target_path.
|
| 50 |
+
"""
|
| 51 |
+
if ignore_path:
|
| 52 |
+
path = Path(ignore_path)
|
| 53 |
+
else:
|
| 54 |
+
path = self.target_path / ".shieldignore"
|
| 55 |
+
|
| 56 |
+
if not path.exists():
|
| 57 |
+
logger.debug(f"No .shieldignore file found at {path}")
|
| 58 |
+
return
|
| 59 |
+
|
| 60 |
+
try:
|
| 61 |
+
with open(path, "r") as f:
|
| 62 |
+
for line_num, line in enumerate(f, 1):
|
| 63 |
+
line = line.strip()
|
| 64 |
+
# Skip empty lines and comments
|
| 65 |
+
if not line or line.startswith("#"):
|
| 66 |
+
continue
|
| 67 |
+
|
| 68 |
+
rule = self._parse_rule(line, line_num)
|
| 69 |
+
if rule:
|
| 70 |
+
self.rules.append(rule)
|
| 71 |
+
|
| 72 |
+
self._stats["rules_loaded"] = len(self.rules)
|
| 73 |
+
logger.info(f"Loaded {len(self.rules)} .shieldignore rules from {path}")
|
| 74 |
+
self._loaded = True
|
| 75 |
+
except IOError as e:
|
| 76 |
+
logger.warning(f"Failed to read .shieldignore: {e}")
|
| 77 |
+
|
| 78 |
+
def _parse_rule(self, line: str, line_num: int) -> Optional[Dict[str, Any]]:
|
| 79 |
+
"""Parse a single .shieldignore rule.
|
| 80 |
+
|
| 81 |
+
Args:
|
| 82 |
+
line: Rule text.
|
| 83 |
+
line_num: Line number in the file.
|
| 84 |
+
|
| 85 |
+
Returns:
|
| 86 |
+
Parsed rule dictionary, or None if invalid.
|
| 87 |
+
"""
|
| 88 |
+
# Support both "type:value" and plain glob patterns
|
| 89 |
+
if ":" in line:
|
| 90 |
+
parts = line.split(":", 1)
|
| 91 |
+
rule_type = parts[0].strip().lower()
|
| 92 |
+
rule_value = parts[1].strip()
|
| 93 |
+
else:
|
| 94 |
+
# Plain pattern is treated as a file/path glob
|
| 95 |
+
rule_type = "path"
|
| 96 |
+
rule_value = line
|
| 97 |
+
|
| 98 |
+
valid_types = {"file", "rule", "category", "severity", "id", "path", "line", "title", "cwe"}
|
| 99 |
+
|
| 100 |
+
if rule_type not in valid_types:
|
| 101 |
+
logger.warning(f"Invalid .shieldignore rule type '{rule_type}' at line {line_num}")
|
| 102 |
+
return None
|
| 103 |
+
|
| 104 |
+
# Compile glob patterns
|
| 105 |
+
compiled_pattern = None
|
| 106 |
+
if rule_type in ("path", "title"):
|
| 107 |
+
# Convert glob to regex
|
| 108 |
+
pattern = re.escape(rule_value)
|
| 109 |
+
pattern = pattern.replace(r"\*", ".*").replace(r"\?", ".")
|
| 110 |
+
try:
|
| 111 |
+
compiled_pattern = re.compile(f"^{pattern}$", re.IGNORECASE)
|
| 112 |
+
except re.error:
|
| 113 |
+
logger.warning(f"Invalid pattern '{rule_value}' at line {line_num}")
|
| 114 |
+
return None
|
| 115 |
+
|
| 116 |
+
rule = {
|
| 117 |
+
"type": rule_type,
|
| 118 |
+
"value": rule_value,
|
| 119 |
+
"pattern": compiled_pattern,
|
| 120 |
+
"line_num": line_num,
|
| 121 |
+
}
|
| 122 |
+
|
| 123 |
+
# Parse compound rules like "line:42:path/to/file.py"
|
| 124 |
+
if rule_type == "line" and ":" in rule_value:
|
| 125 |
+
sub_parts = rule_value.split(":", 1)
|
| 126 |
+
try:
|
| 127 |
+
rule["line_number"] = int(sub_parts[0])
|
| 128 |
+
rule["file_path"] = sub_parts[1]
|
| 129 |
+
except (ValueError, IndexError):
|
| 130 |
+
logger.warning(f"Invalid line rule '{rule_value}' at line {line_num}")
|
| 131 |
+
return None
|
| 132 |
+
|
| 133 |
+
return rule
|
| 134 |
+
|
| 135 |
+
def should_ignore(self, finding: Dict[str, Any]) -> bool:
|
| 136 |
+
"""Check if a finding should be ignored based on .shieldignore rules.
|
| 137 |
+
|
| 138 |
+
Args:
|
| 139 |
+
finding: Finding dictionary.
|
| 140 |
+
|
| 141 |
+
Returns:
|
| 142 |
+
True if the finding should be suppressed.
|
| 143 |
+
"""
|
| 144 |
+
if not self._loaded:
|
| 145 |
+
self.load()
|
| 146 |
+
|
| 147 |
+
for rule in self.rules:
|
| 148 |
+
if self._matches_rule(finding, rule):
|
| 149 |
+
self._stats["findings_filtered"] += 1
|
| 150 |
+
return True
|
| 151 |
+
|
| 152 |
+
return False
|
| 153 |
+
|
| 154 |
+
def _matches_rule(self, finding: Dict[str, Any], rule: Dict[str, Any]) -> bool:
|
| 155 |
+
"""Check if a finding matches a specific ignore rule.
|
| 156 |
+
|
| 157 |
+
Args:
|
| 158 |
+
finding: Finding dictionary.
|
| 159 |
+
rule: Ignore rule dictionary.
|
| 160 |
+
|
| 161 |
+
Returns:
|
| 162 |
+
True if the finding matches the rule.
|
| 163 |
+
"""
|
| 164 |
+
rule_type = rule["type"]
|
| 165 |
+
rule_value = rule["value"]
|
| 166 |
+
|
| 167 |
+
if rule_type == "file":
|
| 168 |
+
# Match file path
|
| 169 |
+
finding_file = finding.get("file", "")
|
| 170 |
+
return finding_file == rule_value or finding_file.endswith(rule_value)
|
| 171 |
+
|
| 172 |
+
elif rule_type == "rule":
|
| 173 |
+
# Match rule ID
|
| 174 |
+
finding_rule = finding.get("rule_id", "")
|
| 175 |
+
return finding_rule == rule_value
|
| 176 |
+
|
| 177 |
+
elif rule_type == "category":
|
| 178 |
+
# Match category
|
| 179 |
+
finding_cat = finding.get("category", "").lower()
|
| 180 |
+
return finding_cat == rule_value.lower()
|
| 181 |
+
|
| 182 |
+
elif rule_type == "severity":
|
| 183 |
+
# Ignore findings at or below this severity
|
| 184 |
+
severity_order = {"INFO": 0, "LOW": 1, "MEDIUM": 2, "HIGH": 3, "CRITICAL": 4}
|
| 185 |
+
finding_sev = severity_order.get(finding.get("severity", "MEDIUM").upper(), 2)
|
| 186 |
+
rule_sev = severity_order.get(rule_value.upper(), 2)
|
| 187 |
+
return finding_sev <= rule_sev
|
| 188 |
+
|
| 189 |
+
elif rule_type == "id":
|
| 190 |
+
# Match exact finding ID
|
| 191 |
+
return finding.get("id", "") == rule_value
|
| 192 |
+
|
| 193 |
+
elif rule_type == "path":
|
| 194 |
+
# Match file path pattern (glob)
|
| 195 |
+
finding_file = finding.get("file", "")
|
| 196 |
+
pattern = rule.get("pattern")
|
| 197 |
+
if pattern is not None:
|
| 198 |
+
try:
|
| 199 |
+
return bool(pattern.match(finding_file))
|
| 200 |
+
except (TypeError, AttributeError):
|
| 201 |
+
return False
|
| 202 |
+
return False
|
| 203 |
+
|
| 204 |
+
elif rule_type == "line":
|
| 205 |
+
# Match specific line in specific file
|
| 206 |
+
if "line_number" in rule and "file_path" in rule:
|
| 207 |
+
return (finding.get("line") == rule["line_number"] and
|
| 208 |
+
finding.get("file", "").endswith(rule["file_path"]))
|
| 209 |
+
return False
|
| 210 |
+
|
| 211 |
+
elif rule_type == "title":
|
| 212 |
+
# Match title pattern (glob)
|
| 213 |
+
finding_title = finding.get("title", "")
|
| 214 |
+
pattern = rule.get("pattern")
|
| 215 |
+
if pattern:
|
| 216 |
+
return bool(pattern.match(finding_title))
|
| 217 |
+
return False
|
| 218 |
+
|
| 219 |
+
elif rule_type == "cwe":
|
| 220 |
+
# Match CWE identifier
|
| 221 |
+
return finding.get("cwe", "") == rule_value
|
| 222 |
+
|
| 223 |
+
return False
|
| 224 |
+
|
| 225 |
+
def filter_findings(self, findings: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
| 226 |
+
"""Filter a list of findings, removing those matching ignore rules.
|
| 227 |
+
|
| 228 |
+
Args:
|
| 229 |
+
findings: List of finding dictionaries.
|
| 230 |
+
|
| 231 |
+
Returns:
|
| 232 |
+
Filtered list of findings.
|
| 233 |
+
"""
|
| 234 |
+
if not self._loaded:
|
| 235 |
+
self.load()
|
| 236 |
+
|
| 237 |
+
if not self.rules:
|
| 238 |
+
return findings
|
| 239 |
+
|
| 240 |
+
filtered = []
|
| 241 |
+
for finding in findings:
|
| 242 |
+
if not self.should_ignore(finding):
|
| 243 |
+
filtered.append(finding)
|
| 244 |
+
else:
|
| 245 |
+
logger.debug(f"Suppressed finding: {finding.get('title', 'unknown')}")
|
| 246 |
+
|
| 247 |
+
removed = len(findings) - len(filtered)
|
| 248 |
+
if removed > 0:
|
| 249 |
+
logger.info(f".shieldignore: Suppressed {removed} findings")
|
| 250 |
+
|
| 251 |
+
return filtered
|
| 252 |
+
|
| 253 |
+
def get_stats(self) -> Dict[str, int]:
|
| 254 |
+
"""Get filtering statistics.
|
| 255 |
+
|
| 256 |
+
Returns:
|
| 257 |
+
Dictionary of filtering statistics.
|
| 258 |
+
"""
|
| 259 |
+
return self._stats.copy()
|
| 260 |
+
|
| 261 |
+
@staticmethod
|
| 262 |
+
def create_template(path: str = ".") -> str:
|
| 263 |
+
"""Create a template .shieldignore file.
|
| 264 |
+
|
| 265 |
+
Args:
|
| 266 |
+
path: Directory to create the file in.
|
| 267 |
+
|
| 268 |
+
Returns:
|
| 269 |
+
Path to the created file.
|
| 270 |
+
"""
|
| 271 |
+
template = """# Shield Agents Ignore File
|
| 272 |
+
# Like .gitignore but for suppressing false positives
|
| 273 |
+
#
|
| 274 |
+
# Rule types:
|
| 275 |
+
# file:path/to/file.py - Ignore all findings in a file
|
| 276 |
+
# rule:SAST-001 - Ignore a specific rule ID
|
| 277 |
+
# category:sql_injection - Ignore all findings of a category
|
| 278 |
+
# severity:LOW - Ignore findings at or below severity
|
| 279 |
+
# id:VulnAgent-3 - Ignore a specific finding by ID
|
| 280 |
+
# path:*.test.* - Ignore findings in matching file paths
|
| 281 |
+
# line:42:path/to/file.py - Ignore finding at specific line
|
| 282 |
+
# title:Assertion* - Ignore findings with matching title
|
| 283 |
+
# cwe:CWE-617 - Ignore findings with matching CWE
|
| 284 |
+
|
| 285 |
+
# Common suppressions for test files
|
| 286 |
+
path:*test*
|
| 287 |
+
path:*spec*
|
| 288 |
+
path:*__pycache__*
|
| 289 |
+
|
| 290 |
+
# Common low-severity suppressions
|
| 291 |
+
# severity:LOW
|
| 292 |
+
# category:information-disclosure
|
| 293 |
+
"""
|
| 294 |
+
dir_path = Path(path)
|
| 295 |
+
dir_path.mkdir(parents=True, exist_ok=True)
|
| 296 |
+
file_path = dir_path / ".shieldignore"
|
| 297 |
+
file_path.write_text(template)
|
| 298 |
+
return str(file_path)
|
shield_agents/utils/__init__.py
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
"""Utility modules for Shield Agents."""
|
shield_agents/utils/crypto.py
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Cryptographic utility functions for Shield Agents."""
|
| 2 |
+
|
| 3 |
+
import hashlib
|
| 4 |
+
import os
|
| 5 |
+
from typing import Optional
|
| 6 |
+
|
| 7 |
+
|
| 8 |
+
def hash_file(file_path: str, algorithm: str = "sha256") -> Optional[str]:
|
| 9 |
+
"""Compute hash of a file for caching/incremental scanning.
|
| 10 |
+
|
| 11 |
+
Args:
|
| 12 |
+
file_path: Path to the file.
|
| 13 |
+
algorithm: Hash algorithm (sha256, md5, sha1).
|
| 14 |
+
|
| 15 |
+
Returns:
|
| 16 |
+
Hex digest string, or None if file cannot be read.
|
| 17 |
+
"""
|
| 18 |
+
try:
|
| 19 |
+
hasher = hashlib.new(algorithm)
|
| 20 |
+
with open(file_path, "rb") as f:
|
| 21 |
+
for chunk in iter(lambda: f.read(8192), b""):
|
| 22 |
+
hasher.update(chunk)
|
| 23 |
+
return hasher.hexdigest()
|
| 24 |
+
except (OSError, IOError, ValueError):
|
| 25 |
+
return None
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
def hash_content(content: str, algorithm: str = "sha256") -> str:
|
| 29 |
+
"""Compute hash of string content.
|
| 30 |
+
|
| 31 |
+
Args:
|
| 32 |
+
content: String content to hash.
|
| 33 |
+
algorithm: Hash algorithm.
|
| 34 |
+
|
| 35 |
+
Returns:
|
| 36 |
+
Hex digest string.
|
| 37 |
+
"""
|
| 38 |
+
hasher = hashlib.new(algorithm)
|
| 39 |
+
hasher.update(content.encode("utf-8"))
|
| 40 |
+
return hasher.hexdigest()
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def generate_id() -> str:
|
| 44 |
+
"""Generate a unique identifier.
|
| 45 |
+
|
| 46 |
+
Returns:
|
| 47 |
+
A random hex string.
|
| 48 |
+
"""
|
| 49 |
+
return os.urandom(16).hex()
|
shield_agents/utils/helpers.py
ADDED
|
@@ -0,0 +1,151 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Helper utility functions for Shield Agents."""
|
| 2 |
+
|
| 3 |
+
import os
|
| 4 |
+
import re
|
| 5 |
+
from pathlib import Path
|
| 6 |
+
from typing import Dict, List, Optional, Tuple
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
def read_file_safe(file_path: str, max_size: int = 1024 * 1024) -> Optional[str]:
|
| 10 |
+
"""Safely read a file with size limits.
|
| 11 |
+
|
| 12 |
+
Args:
|
| 13 |
+
file_path: Path to the file.
|
| 14 |
+
max_size: Maximum file size in bytes.
|
| 15 |
+
|
| 16 |
+
Returns:
|
| 17 |
+
File content as string, or None if file cannot be read.
|
| 18 |
+
"""
|
| 19 |
+
try:
|
| 20 |
+
path = Path(file_path)
|
| 21 |
+
if not path.exists() or not path.is_file():
|
| 22 |
+
return None
|
| 23 |
+
if path.stat().st_size > max_size:
|
| 24 |
+
return None
|
| 25 |
+
with open(file_path, "r", encoding="utf-8", errors="ignore") as f:
|
| 26 |
+
return f.read()
|
| 27 |
+
except (OSError, IOError, UnicodeDecodeError):
|
| 28 |
+
return None
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
def get_file_lines(content: str) -> List[str]:
|
| 32 |
+
"""Split file content into lines.
|
| 33 |
+
|
| 34 |
+
Args:
|
| 35 |
+
content: File content as string.
|
| 36 |
+
|
| 37 |
+
Returns:
|
| 38 |
+
List of lines.
|
| 39 |
+
"""
|
| 40 |
+
return content.split("\n")
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def find_files(
|
| 44 |
+
root_path: str,
|
| 45 |
+
extensions: Optional[List[str]] = None,
|
| 46 |
+
exclude_dirs: Optional[List[str]] = None,
|
| 47 |
+
max_depth: int = 20,
|
| 48 |
+
) -> List[str]:
|
| 49 |
+
"""Find files matching criteria.
|
| 50 |
+
|
| 51 |
+
Args:
|
| 52 |
+
root_path: Root directory to search.
|
| 53 |
+
extensions: File extensions to include (e.g., ['.py', '.js']).
|
| 54 |
+
exclude_dirs: Directory names to exclude.
|
| 55 |
+
max_depth: Maximum directory depth.
|
| 56 |
+
|
| 57 |
+
Returns:
|
| 58 |
+
List of file paths.
|
| 59 |
+
"""
|
| 60 |
+
if exclude_dirs is None:
|
| 61 |
+
exclude_dirs = []
|
| 62 |
+
|
| 63 |
+
found_files = []
|
| 64 |
+
root = Path(root_path)
|
| 65 |
+
|
| 66 |
+
for path in root.rglob("*"):
|
| 67 |
+
# Check depth
|
| 68 |
+
try:
|
| 69 |
+
depth = len(path.relative_to(root).parts)
|
| 70 |
+
except ValueError:
|
| 71 |
+
continue
|
| 72 |
+
if depth > max_depth:
|
| 73 |
+
continue
|
| 74 |
+
|
| 75 |
+
# Check excluded dirs
|
| 76 |
+
if any(exc in path.parts for exc in exclude_dirs):
|
| 77 |
+
continue
|
| 78 |
+
|
| 79 |
+
if path.is_file():
|
| 80 |
+
if extensions is None or path.suffix in extensions:
|
| 81 |
+
found_files.append(str(path))
|
| 82 |
+
|
| 83 |
+
return found_files
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
def extract_code_snippet(content: str, line_number: int, context: int = 3) -> str:
|
| 87 |
+
"""Extract a code snippet around a specific line.
|
| 88 |
+
|
| 89 |
+
Args:
|
| 90 |
+
content: Full file content.
|
| 91 |
+
line_number: Target line number (1-based).
|
| 92 |
+
context: Number of context lines before and after.
|
| 93 |
+
|
| 94 |
+
Returns:
|
| 95 |
+
Code snippet as string.
|
| 96 |
+
"""
|
| 97 |
+
lines = content.split("\n")
|
| 98 |
+
start = max(0, line_number - 1 - context)
|
| 99 |
+
end = min(len(lines), line_number + context)
|
| 100 |
+
snippet_lines = lines[start:end]
|
| 101 |
+
|
| 102 |
+
result = []
|
| 103 |
+
for i, line in enumerate(snippet_lines, start=start + 1):
|
| 104 |
+
marker = ">>>" if i == line_number else " "
|
| 105 |
+
result.append(f"{marker} {i:4d} | {line}")
|
| 106 |
+
|
| 107 |
+
return "\n".join(result)
|
| 108 |
+
|
| 109 |
+
|
| 110 |
+
def truncate_string(s: str, max_length: int = 200) -> str:
|
| 111 |
+
"""Truncate a string to a maximum length.
|
| 112 |
+
|
| 113 |
+
Args:
|
| 114 |
+
s: Input string.
|
| 115 |
+
max_length: Maximum length.
|
| 116 |
+
|
| 117 |
+
Returns:
|
| 118 |
+
Truncated string with ellipsis if needed.
|
| 119 |
+
"""
|
| 120 |
+
if len(s) <= max_length:
|
| 121 |
+
return s
|
| 122 |
+
return s[:max_length - 3] + "..."
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
def calculate_risk_score(findings: List[Dict]) -> int:
|
| 126 |
+
"""Calculate an overall risk score from findings.
|
| 127 |
+
|
| 128 |
+
Args:
|
| 129 |
+
findings: List of finding dictionaries.
|
| 130 |
+
|
| 131 |
+
Returns:
|
| 132 |
+
Risk score from 0-100.
|
| 133 |
+
"""
|
| 134 |
+
if not findings:
|
| 135 |
+
return 0
|
| 136 |
+
|
| 137 |
+
severity_weights = {
|
| 138 |
+
"CRITICAL": 25,
|
| 139 |
+
"HIGH": 15,
|
| 140 |
+
"MEDIUM": 8,
|
| 141 |
+
"LOW": 3,
|
| 142 |
+
"INFO": 1,
|
| 143 |
+
}
|
| 144 |
+
|
| 145 |
+
total = sum(
|
| 146 |
+
severity_weights.get(f.get("severity", "MEDIUM").upper(), 5)
|
| 147 |
+
for f in findings
|
| 148 |
+
)
|
| 149 |
+
|
| 150 |
+
# Cap at 100
|
| 151 |
+
return min(total, 100)
|
tests/__init__.py
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
"""Tests for Shield Agents."""
|
tests/test_cache.py
ADDED
|
@@ -0,0 +1,99 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Tests for the caching system."""
|
| 2 |
+
|
| 3 |
+
import os
|
| 4 |
+
import sys
|
| 5 |
+
import tempfile
|
| 6 |
+
import unittest
|
| 7 |
+
|
| 8 |
+
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
|
| 9 |
+
|
| 10 |
+
from shield_agents.cache import ScanCache
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
class TestScanCache(unittest.TestCase):
|
| 14 |
+
"""Test the scan cache."""
|
| 15 |
+
|
| 16 |
+
def setUp(self):
|
| 17 |
+
self.tmpdir = tempfile.mkdtemp()
|
| 18 |
+
self.cache = ScanCache(cache_dir=os.path.join(self.tmpdir, ".shield-cache"))
|
| 19 |
+
|
| 20 |
+
def tearDown(self):
|
| 21 |
+
import shutil
|
| 22 |
+
shutil.rmtree(self.tmpdir, ignore_errors=True)
|
| 23 |
+
|
| 24 |
+
def test_file_changed_detection(self):
|
| 25 |
+
# Create a file
|
| 26 |
+
test_file = os.path.join(self.tmpdir, "test.py")
|
| 27 |
+
with open(test_file, "w") as f:
|
| 28 |
+
f.write("print('hello')")
|
| 29 |
+
|
| 30 |
+
# First check - should be "changed" (not in cache)
|
| 31 |
+
self.assertTrue(self.cache.is_file_changed(test_file))
|
| 32 |
+
|
| 33 |
+
# Update cache
|
| 34 |
+
self.cache.update_file(test_file, [{"title": "test", "severity": "LOW"}])
|
| 35 |
+
self.cache.save()
|
| 36 |
+
|
| 37 |
+
# After caching - should NOT be changed
|
| 38 |
+
self.assertFalse(self.cache.is_file_changed(test_file))
|
| 39 |
+
|
| 40 |
+
def test_modified_file_detection(self):
|
| 41 |
+
test_file = os.path.join(self.tmpdir, "test.py")
|
| 42 |
+
with open(test_file, "w") as f:
|
| 43 |
+
f.write("print('hello')")
|
| 44 |
+
|
| 45 |
+
self.cache.update_file(test_file, [])
|
| 46 |
+
self.cache.save()
|
| 47 |
+
|
| 48 |
+
# Modify the file
|
| 49 |
+
with open(test_file, "w") as f:
|
| 50 |
+
f.write("print('modified')")
|
| 51 |
+
|
| 52 |
+
# Should now be detected as changed
|
| 53 |
+
self.assertTrue(self.cache.is_file_changed(test_file))
|
| 54 |
+
|
| 55 |
+
def test_get_cached_findings(self):
|
| 56 |
+
test_file = os.path.join(self.tmpdir, "test.py")
|
| 57 |
+
with open(test_file, "w") as f:
|
| 58 |
+
f.write("x = 1")
|
| 59 |
+
|
| 60 |
+
findings = [{"title": "SQL Injection", "severity": "CRITICAL"}]
|
| 61 |
+
self.cache.update_file(test_file, findings)
|
| 62 |
+
self.cache.save()
|
| 63 |
+
|
| 64 |
+
cached = self.cache.get_cached_findings(test_file)
|
| 65 |
+
self.assertEqual(len(cached), 1)
|
| 66 |
+
self.assertEqual(cached[0]["title"], "SQL Injection")
|
| 67 |
+
|
| 68 |
+
def test_get_changed_files(self):
|
| 69 |
+
f1 = os.path.join(self.tmpdir, "a.py")
|
| 70 |
+
f2 = os.path.join(self.tmpdir, "b.py")
|
| 71 |
+
|
| 72 |
+
with open(f1, "w") as f:
|
| 73 |
+
f.write("a = 1")
|
| 74 |
+
with open(f2, "w") as f:
|
| 75 |
+
f.write("b = 2")
|
| 76 |
+
|
| 77 |
+
# Cache only f1
|
| 78 |
+
self.cache.update_file(f1, [])
|
| 79 |
+
self.cache.save()
|
| 80 |
+
|
| 81 |
+
changed = self.cache.get_changed_files([f1, f2])
|
| 82 |
+
self.assertIn(f2, changed)
|
| 83 |
+
self.assertNotIn(f1, changed)
|
| 84 |
+
|
| 85 |
+
def test_clear_cache(self):
|
| 86 |
+
test_file = os.path.join(self.tmpdir, "test.py")
|
| 87 |
+
with open(test_file, "w") as f:
|
| 88 |
+
f.write("x = 1")
|
| 89 |
+
|
| 90 |
+
self.cache.update_file(test_file, [])
|
| 91 |
+
self.cache.save()
|
| 92 |
+
self.cache.clear()
|
| 93 |
+
|
| 94 |
+
# After clear, file should be "changed"
|
| 95 |
+
self.assertTrue(self.cache.is_file_changed(test_file))
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
if __name__ == "__main__":
|
| 99 |
+
unittest.main()
|