Evaluating and buying AI security tools
Many security products now include AI. Ask the right questions, test on your own data, and compare against a baseline before you buy.
In 60 seconds
Evaluating and buying AI security tools
Many security products now include AI. Ask the right questions, test on your own data, and compare against a baseline before you buy.
Questions to ask vendors
| Area | Questions |
|---|---|
| Data residency | Where is our data stored and processed? Can it stay in a region we choose? |
| Retention | How long are prompts, logs and outputs kept? Can we delete them? |
| Training | Is our data used to train or improve models? Is that off by default? Is it in the contract? |
| Evaluation | Can we test on our own data during a pilot? What results did others see, and how was that measured? |
| False positives | What false-positive and missed-detection rates can we expect in an environment like ours? |
| Integrations | Does it work with our SIEM, ticketing, identity and endpoint tools? Read-only or with actions? |
| Audit logs | Are all AI actions and user approvals logged, and can we export those logs? |
| Controls | Can we require approval for actions, limit permissions, and switch features off? |
| Security of the tool | How do they protect against prompt injection and data leaks between customers? |
| Pricing | How is it priced (per user, per data volume, per AI query)? What happens if use grows? |
Run a fair pilot
- 1
Set a baseline
Measure today's numbers first: time to triage, false positives, missed cases, analyst hours. - 2
Define success
Agree targets before the pilot, not after. - 3
Use real cases
Test on your own alerts and data, including past incidents with known answers. - 4
Compare fairly
Same types of cases, same time period, with and without the tool. - 5
Ask the analysts
Did it help? What did they have to fix? Did they trust it too much? - 6
Decide and document
Buy, extend the pilot, or stop, with reasons written down.
- Decided after one impressive demo
- No numbers from before the pilot
- Data terms not checked
- Pilot on your own data with a baseline
- Success targets agreed up front
- Data, retention and training terms in the contract
Watch and read more
Lab
A vendor evaluation scorecard and a one-page pilot plan.
The problem
Imagine your SOC is choosing between two AI triage tools. You do not need real vendors; use "Tool A" and "Tool B".
- Build a scorecard (a table or spreadsheet) with at least 10 criteria from this lesson. Give each a weight (for example 1 to 3) and a scoring guide.
- Mark which criteria are "must have" (a tool fails if it does not meet them).
- Write a one-page pilot plan: baseline measures, success targets, test cases, length, and who decides.
- List 5 questions you would send to both vendors in writing.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Tool A is cheaper but keeps prompts for a long time. Tool B costs more but lets you set retention. How do you decide?Reveal
Check what data the prompts will contain. If they include personal or customer data, long retention may conflict with your data rules and contracts, which could make Tool A unacceptable at any price. If the risk is acceptable after legal review, compare total cost and results. Treat must-haves as pass or fail before comparing price.
Q2Why can a pilot on only easy, old alerts give a misleading result?Reveal
Easy cases make every tool look good, and old cases may resemble examples the vendor tuned for. Real value shows on hard, messy, recent cases, including ones where the right answer is "this is a real attack". Include a mix, and include cases with known answers so you can measure missed detections.
Questions people ask
Why test on our own data if the vendor has good benchmark results?
Every environment is different: different logs, users, applications and normal behaviour. A tool that does well on someone else's data may produce many false positives on yours. Only your own pilot tells you.
Is it okay to give a vendor real data for a pilot?
Only after legal and privacy review, with a signed agreement covering storage, retention, deletion and training. Share the minimum needed.
How long should a pilot be?
Long enough to see normal variation and a fair number of real cases. The right length depends on your alert volume. Agree the length and success targets before starting.
What if analysts love the tool but the numbers are flat?
Look deeper. It may reduce stress or help new analysts, which has value. But also check whether people trust it too much. Decide with both the numbers and the feedback, and write down why.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.