Overview
- Direct https://github.com/promptfoo/promptfoo
- OS Linux, Mac (Homebrew)
Promptfoo sits at the intersection of LLM evaluation, adversarial testing, prompt-injection testing, agent security, and RAG validation.
Think of Promptfoo as pytest + benchmark framework + security scanner for LLM applications.
It lets you define:
INPUTS
│
├── Prompt A ──► Model A
├── Prompt A ──► Model B
├── Prompt B ──► Model A
└── Prompt B ──► Model B
│
▼
Assertions / Graders
│
▼
PASS / FAIL / SCORE
And for security:
Promptfoo
│
├── Prompt injection
├── Jailbreaks
├── Data leakage
├── RAG poisoning
├── Excessive agency
├── System prompt extraction
├── Harmful-content tests
└── Custom security policies
│
▼
Your LLM / Agent / RAG
Promptfoo specifically supports automated evaluation of prompts, models, RAG pipelines and agents, alongside automated red teaming and CI/CD integration.
Installation
- Based on CachyOS/Arch Linux
- Promptfoo currently requires Node.js 22.22.0 or newer.
Update CachyOS and install Node.js 24 LTS + npm
$ sudo pacman -Syu
$ sudo pacman -S nodejs-lts-krypton npm
$ node --version
$ npm --version
Install some useful development dependencies
$ sudo pacman -S --needed \
git \
python \
python-pip \
jq \
curl \
ripgrep \
base-devel
Configure npm for our user
$ mkdir -p ~/.local/bin
$ mkdir -p ~/.local/lib
$ npm config set prefix "$HOME/.local"
$ fish_add_path -m ~/.local/bin
# Or if you use Bash
$ echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
$ source ~/.bashrc
# Or if you use Zsh
$ echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc
$ source ~/.zshrc
$ npm config get prefix
Install Promptfoo
$ npm install -g promptfoo
$ promptfoo --version
$ which promptfoo
Check Promptfoo
$ promptfoo --help
$ promptfoo --version
$ promptfoo redteam --help
$ promptfoo redteam plugins
Create our Promptfoo workspace
$ mkdir -p ~/labs/promptfoo
$ cd ~/labs/promptfoo
$ promptfoo init
$ cd ~/labs
$ promptfoo init --example getting-started
$ cd getting-started
$ ls -la
Configure OpenAI (If you have an OpenAI API key)
$ export OPENAI_API_KEY="YOUR_API_KEY"
$ test -n "$OPENAI_API_KEY" && echo "OPENAI_API_KEY is set"
For persistence, we recommend using a .env file inside the project, rather than putting the key directly into YAML:
$ nano .env
Put:
OPENAI_API_KEY=YOUR_API_KEY
Then:
$ chmod 600 .env
And make sure Git doesn’t accidentally commit it:
$ echo ".env" >> .gitignore
Promptfoo supports .env files and the --env-file option.
First evaluation
$ cd ~/labs/getting-started
$ promptfoo eval
$ promptfoo view
This gives you the web-based evaluation interface.
Install Ollama
For our security research, we also install Ollama so we can test local models without sending everything to an API provider.
$ sudo pacman -S ollama
$ sudo systemctl enable --now ollama
$ systemctl status ollama
$ ollama --version
$ ollama pull llama3.3
$ ollama list
$ ollama run llama3.3
$ /bye
Test OpenAI + Ollama side-by-side
Create
$ nano promptfooconfig.yaml
For example:
description: WE ARE THE BUG - basic LLM evaluation
prompts:
- |
You are a cybersecurity research assistant.
Answer the following question accurately and concisely:
{{question}}
providers:
- openai:gpt-5
- ollama:chat:llama3.3
tests:
- vars:
question: "What is Kerberoasting?"
assert:
- type: contains
value: "Kerberos"
- vars:
question: "What is prompt injection?"
assert:
- type: contains
value: "instruction"
- vars:
question: "Explain the principle of least privilege."
assert:
- type: contains
value: "privilege"
Run
$ promptfoo eval
Currently we have:
SAME TEST
│
┌─────────┴─────────┐
▼ ▼
GPT-5 Llama 3.3
│ │
└─────────┬─────────┘
▼
Assertions
│
▼
Results
That’s our first model benchmark.
Add Anthropic
If you have an Anthropic API key:
$ nano .env
Add:
ANTHROPIC_API_KEY=YOUR_API_KEY
Then add another provider:
providers:
- openai:gpt-5
- anthropic:messages:claude-sonnet-5
- ollama:chat:llama3.3
And then let’s evaluate:
$ promptfoo eval
So we have compare:
TEST SUITE
│
┌─────────────┼─────────────┐
▼ ▼ ▼
GPT Claude Ollama
│ │ │
└─────────────┼─────────────┘
▼
Evaluation
Plugins
Now we install/use some interesting plugins.
Promptfoo red-team plugins are part of the red-team framework.
List them:
$ promptfoo redteam plugins
For our lab. we can initially concentrate on:
prompt-injection
jailbreak
pii
excessive-agency
hijacking
hallucination
overreliance
RAG poisoning
data leakage
Obtain the IDs supported by our installed version (exact plugin IDs can change between Promptfoo versions):
$ promptfoo redteam plugins
First red-team project
$ cd ~/labs
$ promptfoo redteam init ai-security-lab --no-gui
$ cd ai-security-lab
$ ls -la
Follow the workflow.
Run the red-team scan:
$ promptfoo redteam run
Read the report:
$ promptfoo redteam report
We can explicitly select plugins.
For example:
$ promptfoo redteam generate \
--plugins "hijacking,excessive-agency,hallucination,pii"
$ promptfoo redteam run
Do not forget that the exact plugin IDs available to you should be checked first:
$ promptfoo redteam plugins
For many tests, we recommand to create a similar directory structure:
~/labs/promptfoo/
│
├── prompts/
│ ├── system-v1.txt
│ ├── system-v2.txt
│ └── security-agent.txt
│
├── evaluations/
│ ├── accuracy/
│ ├── regression/
│ └── model-comparison/
│
├── rag/
│ ├── retrieval/
│ ├── grounding/
│ └── poisoning/
│
├── agents/
│ ├── tool-use/
│ ├── permissions/
│ └── trajectories/
│
├── redteam/
│ ├── injection/
│ ├── jailbreak/
│ ├── leakage/
│ └── agency/
│
└── README.md
- TARGET is different than ATTACK GENERATOR / GRADER
- Promptfoo’s current red-team implementation can use your configured OpenAI key for attack generation/grading.
- If you don’t provide one, its remote generation service can be used. -Target evaluation itself runs locally.
For example:
Promptfoo
│
┌──────────┴──────────┐
▼ ▼
Attack model Target model
│ │
▼ ▼
Generate attack Your AI
│ │
└──────────┬──────────┘
▼
Grader
Optional
Python agent/RAG integration
Because you’re likely to eventually test Python-based agents/RAG pipelines, make sure Python is available:
$ python --version
Create a virtual environment and then activate it:
$ python -m venv ~/labs/promptfoo/.venv
$ source ~/labs/promptfoo/.venv/bin/activate
If needed, Install/Upgrade pip:
$ pip install --upgrade pip
Then you can install whatever RAG/agent framework you’re testing without polluting the CachyOS system Python.
Promptfoo supports custom Python providers such as:
providers:
- file://provider.py
Example of eventual architecture:
Promptfoo
│
▼
provider.py
│
▼
Your Python Agent
│
├── LLM
├── RAG
├── Vector DB
└── Tools
Git repository
$ cd ~/labs/promptfoo
$ git init
Create:
$ nano .gitignore
Put:
.env
*.log
node_modules/
.promptfoo/
.venv/
__pycache__/
Then:
$ git add .
$ git commit -m "Initial Promptfoo AI security lab"
Some other examples
If all above works, then we are ready for the Promptfoo evaluation + model comparison + RAG testing + agent testing + AI red-team workflow.
We will continue to update here with practical examples and use cases…