Skip to content
Counsel.Directory

Prompt Testing for Law Firms

Software testing methods applied to legal AI prompts, so that your firm knows how reliable a prompt is before a junior lawyer uses it.

Get in TouchView Methodology

The Problem

For Knowledge Managers

• A set of prompts can take months of work to write

• No record of how often a prompt gives the same answer

• No agreed method for testing a prompt

For Junior Lawyers

• Juniors need AI most, but cannot check its output

• Seniors can check it, but need AI less

• Every AI output waits for a senior lawyer's time

• Risk of invented content and errors in client work

• Need checklists and clear rules on when to escalate

The Approach

Software quality assurance applied to legal AI prompts: the testing principles used in software development, adapted for legal work.

1

Systematic Testing

Run each prompt 20 times on real and synthetic data. Measure consistency (the same input gives the same output) and accuracy (the output against a lawyer's review).

2

Five Validation Layers

The AI checks its own answer, scores its confidence, and is checked by rules written in code. The firm's own context is added, and each task is graded by complexity, so a junior lawyer knows when to escalate.

3

Automation

Automated test runs, quality dashboards, version control for prompts and regression testing, so that less of the testing is done by hand.

4

Knowledge Transfer

A methodology handbook, team training and documentation, so that your team can maintain the prompts after the engagement ends.

Service Packages

Starting Point
Rapid Pilot
2 weeks

Test three prompts and show the results before and after improvement.

  • Three prompts, each run 20 times
  • Before/after comparison with data
  • A demonstration of the testing method
  • Recommendations report
Prompt Library
Library Kickstart
8 weeks

Build and test 50 prompts for your practice area.

  • 50 prompts, each tested the same way
  • Quality scores & benchmarks
  • Version control system setup
  • Team training & documentation
Ongoing Support
Testing-as-a-Service
Monthly retainer

Up to 10 prompts tested each month.

  • Up to 10 prompts tested per month
  • A test report for each prompt
  • Prompts improved as models change
On Request
Prompt Designer Retainer
Monthly retainer

Prompt design when you need it, without a full-time hire.

  • Monthly design review sessions
  • Slack/Teams direct access
  • Best practices training

The price depends on the size of your firm and the work involved. Email hello@counsel.directory for a quote.

What the Work Takes

Legal Work

How law firms work, how reliable legal work has to be, and the duties of professional conduct.

AI and Prompt Engineering

How large language models work and fail, and how to write prompts for them.

Software Testing

Quality assurance, version control and regression testing, applied to prompts.

Testing a legal prompt takes all three: knowledge of the legal work, of the AI model, and of how software is tested.

Get in Touch

Interested in prompt testing for your firm? Describe what you need.

Pilot projects are open to firms in England and Wales. No prompt in this library has published test results yet. When one has, its page will show them.

Email: hello@counsel.directory

Free Resources

The prompts and the methodology on this site are free to read and use.

Browse Free PromptsRead the Methodology