Insights / AI Risk Management and Quality Assurance

The Hidden Risks of Uploading Medical Records Into General-Purpose AI Tools

By Breen LNC.

Short answer: Uploading records into a general AI tool can expose confidential health information under terms you have not read. Bar guidance warns that some consumer products offer no contractual confidentiality, vendor coverage can vary feature by feature, and accuracy is not guaranteed. Check the contract and your duties first.

Confidentiality comes first

Washington’s 2025 advisory opinion says some consumer AI products don’t include contractual assurances of confidentiality, which it says poses an unreasonable risk to client information. Entering identifiable client information into such a product raises serious concerns under the confidentiality rule. Depending on sensitivity, the lawyer may need to consult the client and sometimes get informed consent.

Arizona’s guidance quotes ABA Formal Opinion 512 on why: because many self-learning tools could disclose information relating to a representation, a client’s informed consent is required before entering that information into the tool. Arizona adds, in its own words, that a free AI tool “may offer little or no protection of the data,” and that an option not to use your data for training doesn’t completely eliminate the risk. Medical records are about as sensitive as client information gets.

What the vendors’ own pages show

Vendor terms are specific, and they change. Treat these as examples of why you have to read the current page for the exact product you’re using.

  • Google’s Workspace privacy hub says customer data isn’t used to train its generative AI models without permission, and also states that some products, Gemini Notebook and Gemini in Chrome, aren’t covered by its Business Associate Agreement. That page covers business accounts, not personal ones.
  • Microsoft’s Copilot documentation says commercial prompts and responses aren’t used to train its foundation models. It also warns, “The responses that generative AI produces aren’t guaranteed to be 100% factual.”

The lesson is that “the company has an enterprise plan” doesn’t answer the question. Which account, which feature and which contract all matter. Staff using a personal account at home may be outside anything your firm negotiated.

Where HIPAA fits

A business associate generally is someone who creates, receives, maintains or transmits protected health information for a covered entity’s regulated function. HHS guidance lists cloud providers among its examples and requires written assurances. Whether HIPAA applies to your firm or to a given vendor depends on the facts, including how you received the records. Don’t assume either way, and ask counsel. Your ethical duty of confidentiality applies regardless.

What to do this week

  1. Ask your team what they’re already using. Assume someone has tried it.
  2. Write a short rule on what may and may not go into which tools.
  3. Read the terms for every approved tool, or have someone who has analyzed them explain them.
  4. Check who can access inputs, how long they’re kept, and whether they train models.
  5. Ask for security evidence, and don’t accept a marketing page as an answer.

In Breen’s workflow, real patient records go in only after the agreements are signed, and a nurse reviews each entry.

Frequently asked questions

Can I upload client medical records into ChatGPT or a similar tool?

Be very careful. Bar guidance warns that some consumer products have no contractual confidentiality assurance and that client consent may be required. Check the terms for your exact account and consult your counsel. We did not verify any particular vendor’s consumer terms.

Does a vendor BAA cover every AI feature?

Not necessarily. Google, for example, states that some of its Workspace products are not covered by its BAA. Check coverage for each feature you intend to use.

Sources

General information for legal professionals. This is not legal or medical advice.