Insights / AI Risk Management and Quality Assurance

How to Evaluate an AI Medical Record Review Platform Before Your Firm Invests

By Breen LNC.

Short answer: Evaluate a platform on your own closed files, not its demo. Measure what it misses as well as what it gets wrong. Then read the data terms, ask for independent security evidence, find out exactly who reviews the output, and confirm you can get your work back out if you leave.

Test it on a case you know

A demo shows the vendor’s best file. Instead, run one of your closed cases and compare the output against what your team found. Count errors and omissions separately. NIST’s AI RMF core says AI systems should be “tested before their deployment and regularly while in operation,” with risks tracked over time. That’s a voluntary framework and not a buyer’s guide, but it gives you sensible language to use.

Ask how the vendor measures accuracy. Stanford researchers found providers’ claims about legal research tools were overstated when tested independently, though those were different products at an earlier date. Ask for the method, the sample and whether anyone independent checked it.

Questions on data

  • Is our data used to train any model?
  • Who can see it, including subcontractors?
  • How long is it kept, and how is it deleted?
  • Where is it stored?
  • Will you put this in the contract?

Washington’s advisory opinion tells lawyers to read the terms to see whether data is used for another purpose. If patient information is involved, ask about business associate agreements. HHS says downstream subcontractors handling this data are business associates too, where HIPAA applies, so ask about the whole chain.

Questions on security

A SOC 2 report is one piece of evidence. AICPA describes it as an examination performed by a licensed, independent CPA, evaluating controls relevant to security, availability, processing integrity, confidentiality or privacy. Ask to see it, not just a logo, and check its dates and scope. It doesn’t guarantee accuracy or HIPAA compliance.

Questions on review

  • Does a licensed clinician review every entry, or a sample?
  • Whose license is it, and what’s their background?
  • Can a reviewer see the source beside each entry?
  • Is there a second reviewer?
  • What happens to pages the software can’t read?

Questions on traceability

Does each fact link to a stable page number? Does the software confirm the quoted line appears on that page? Are changes logged by person and time?

Questions on leaving

Can you export your records, chronologies and logs in usable formats? What happens to your data after you cancel? Lock-in tends to look harmless until you want out.

Watch for these

Be wary of a promise of “no hallucinations,” accuracy percentages without method, vague “human in the loop” language, or reluctance to run your own file. A good vendor should welcome the test.

Frequently asked questions

How should a law firm test AI medical review software?

Run a closed case your team knows well and count errors and omissions separately. Ask how the vendor measures accuracy and whether anyone independent has checked it.

Does a SOC 2 report mean an AI tool is safe to use?

It is useful evidence about controls, performed by an independent CPA, but it is not a guarantee of security, HIPAA compliance or AI accuracy. Check its dates and scope.

Sources

General information for legal professionals. This is not legal or medical advice.