By the Shadow AI Policy team
HIPAA was written for filing cabinets and fax machines — but regulators are applying it to ChatGPT, Gemini, and every other AI tool your staff is already using.
This post covers the five things covered entities and business associates need to get right when AI tools touch protected health information: whether your AI vendor needs a BAA, what that BAA must include, how to classify which PHI inputs are ever permissible, what an AI-related breach notification looks like under 45 CFR § 164.400–414, and how the minimum necessary standard constrains what your employees can paste into a prompt.
If an AI vendor processes PHI on your behalf — even to improve a summary, transcribe a call, or generate a clinical note draft — they are almost certainly a business associate under HIPAA, and using them without a signed BAA is a HIPAA violation starting with the first input. Don't wait for your annual audit to fix this.
Under HIPAA's Privacy Rule at 45 CFR § 160.103, a business associate is any person or entity that creates, receives, maintains, or transmits PHI on behalf of a covered entity. The test isn't whether the vendor calls themselves a healthcare company — it's whether PHI touches their systems as part of the work they're doing for you.
When a nurse pastes a patient summary into ChatGPT to draft a discharge letter, OpenAI's systems are receiving and processing PHI. When a biller uploads a claims spreadsheet into Google Gemini to spot coding errors, same situation. The vendor becomes a business associate the moment PHI enters their environment — regardless of whether you intended that or whether the vendor knew it contained PHI.
There's a narrow exception: a vendor who processes PHI only incidentally, without any ability to access or use it (the "conduit exception" under HHS guidance), may not be a business associate. But AI tools don't qualify for this — they're specifically analyzing the content you submit. If the AI reads it to do its job, the vendor is a business associate.
This matters because using a non-BAA vendor to process PHI isn't a gray area. It's a Privacy Rule violation under 45 CFR § 164.502(e), which prohibits covered entities from disclosing PHI to a business associate without a compliant agreement in place.
A Business Associate Agreement (BAA) is a HIPAA-required contract governed by 45 CFR § 164.504(e). It must cover specific obligations — not just be a generic data processing addendum with HIPAA mentioned somewhere. For AI vendors specifically, the BAA needs to address:
Some AI vendors offer HIPAA-ready tiers with BAAs. Microsoft 365 Copilot in enterprise configurations includes a BAA through Microsoft's Health Data Services agreement. ChatGPT Enterprise offers a BAA — but the standard ChatGPT free and Plus tiers do not. Gemini for Google Workspace (Enterprise and Education Plus) includes a BAA; the consumer Gemini product does not. For a current comparison of which tools offer BAAs, see our HIPAA-compliant AI tools guide.
If a vendor won't sign a BAA, you have two options: don't use the tool with PHI, or don't use the tool at all if employees can't reliably keep PHI out of it. "We asked staff not to use it for patient data" is not a HIPAA defense.
Not all AI tool deployments carry the same risk profile. The practical framework is to classify your AI tools into three tiers based on their data handling guarantees, then match each tier to the types of inputs that are permissible.
| Tool Tier | Examples | BAA Available? | PHI Permitted? | Conditions |
|---|---|---|---|---|
| Tier 1 — HIPAA-Ready Enterprise | ChatGPT Enterprise, Microsoft 365 Copilot, Gemini for Workspace Enterprise | Yes | Yes, with controls | BAA signed, minimum necessary enforced, staff trained |
| Tier 2 — Paid/Team (No BAA) | ChatGPT Plus, Claude (consumer), Notion AI | No | No | PHI must never enter these tools; block access if employees can't comply |
| Tier 3 — Free/Consumer | ChatGPT (free), Meta AI, Perplexity, xAI Grok | No | No | Block at the network level; do not rely on policy alone |
Even within Tier 1 tools, PHI inputs should be the minimum necessary to accomplish the task. Passing a full patient record when only a diagnosis code is needed creates unnecessary exposure — and violates the minimum necessary principle discussed below.
For employees using consumer AI tools from personal devices, the risk is harder to control technically, which is why your acceptable use policy needs to address personal device use explicitly. A solid AI acceptable use policy should classify tool tiers and prohibited input types in writing, not leave it to employee judgment.
Sending PHI to an AI vendor without a BAA is a disclosure that triggers HIPAA's Breach Notification Rule under 45 CFR §§ 164.400–414 — unless you can demonstrate a low probability of compromise using the four-factor risk assessment defined in 45 CFR § 164.402. That assessment requires evaluating: the nature and extent of the PHI involved, who accessed it, whether it was actually acquired or viewed, and the extent to which risk has been mitigated.
In practice, proving "low probability of compromise" for PHI submitted to a commercial AI tool is very difficult. These platforms log inputs, may use them in training pipelines (depending on tier and settings), and route data through multiple infrastructure layers — all of which undercut a risk mitigation argument. Assume that an unauthorized PHI disclosure to an AI vendor is a reportable breach until your legal counsel says otherwise.
The notification timeline is strict: covered entities must notify affected individuals without unreasonable delay and no later than 60 calendar days after discovery (45 CFR § 164.404). If the breach affects 500 or more residents of a state or jurisdiction, HHS must be notified simultaneously and prominent media outlets in that area must be notified. Breaches affecting fewer than 500 individuals can be logged and reported to HHS annually, but the individual notification requirement still applies promptly.
The practical implication: if you discover that an employee has been pasting patient records into a non-BAA AI tool for the past three months, you cannot simply revoke access and move on. You need to conduct a breach risk assessment, document it, and — depending on the outcome — notify individuals and HHS. Build AI-related breach scenarios into your incident response plan now, before you're running the assessment under pressure.
Under 45 CFR § 164.502(b), covered entities must make reasonable efforts to limit PHI to the minimum necessary to accomplish the intended purpose. This applies to every disclosure — including inputs into AI tools.
In practical terms, the minimum necessary standard means your AI use policy needs to specify what data employees are permitted to include in prompts, not just which tools they're allowed to use. Vague guidance like "don't share unnecessary patient information" isn't operationally useful. Staff need concrete rules.
Here's what minimum necessary looks like in AI-specific guidance:
The minimum necessary standard also applies to system prompts and retrieval-augmented generation (RAG) setups where AI tools are connected to internal databases. If your AI system can retrieve and surface full patient records in response to a broad query, you've likely violated minimum necessary even if no human manually entered the data. Review AI system architecture — not just human input behavior — against this standard.
For help assessing which AI tools your organization should permit and drafting input restriction rules, generate a tailored policy kit that covers PHI-specific guidance alongside your broader acceptable use policy.
If your organization handles PHI and you don't yet have AI-specific policy language, these are the non-negotiable items to add immediately:
For context on why employees use unsanctioned AI tools in the first place — and how shadow AI spreads in organizations that don't have clear policies — see our overview of what shadow AI is and why it happens.
About Shadow AI Policy: We build AI acceptable use policy tools for HR and operations teams at 50–500 person companies. We publish guides on shadow AI, acceptable use policies, and AI governance, updated as regulations and AI tools change.
Treat it as a potential breach immediately. Under 45 CFR § 164.402, you must conduct a four-factor risk assessment to determine whether there's a low probability that the PHI was compromised. If you can't make that case — and with commercial AI tools, it's usually hard to — you're looking at breach notification obligations: individuals within 60 days, HHS within 60 days, and potentially media notice if more than 500 people in a jurisdiction are affected. Document the assessment regardless of the outcome.
Yes, if the embedded AI processes PHI. The key question is whether a new subprocessor — the AI model provider — now has access to PHI that wasn't contemplated in the original vendor agreement. Many EHR vendors are adding AI features powered by third-party models (OpenAI, Google, Anthropic). Check whether your existing BAA with the EHR vendor covers these new AI subprocessors, and whether the EHR vendor has a BAA with the underlying AI provider. If not, the chain is broken and your covered entity is exposed.
Yes — if the data is truly de-identified under HIPAA's standards. HIPAA's Safe Harbor method under 45 CFR § 164.514(b) requires removing 18 specific identifiers (name, geographic data below state level, dates other than year, phone numbers, email, SSN, MRN, and others). If those identifiers are genuinely removed and there's no reasonable basis to believe the data could be re-identified, it's no longer PHI and HIPAA's BAA requirement doesn't apply. But be careful: partially de-identified data — for example, removing names but keeping diagnosis, ZIP code, and exact date of service — can still be re-identifiable and doesn't qualify for Safe Harbor.
At minimum, annually — but realistically, any time a new AI tool gains significant adoption in your organization, a vendor changes its data processing terms, or a tool you've approved loses its BAA offering. AI vendor terms change frequently: a tool that offered a BAA under enterprise terms last year may have restructured its product tiers. Assign a specific person or role to own the approved tools list and set a calendar reminder to verify BAA status every six months. Don't rely on the original procurement review to stay current.
Free tool: Check what an AI tool does with your data → Training on your content, retention, admin controls and BAA availability for 26 major tools.
Tailored to your industry and the AI tools your team uses. Free preview, then $149/mo to keep it current as the rules and vendor terms change — or $79 for a one-time snapshot.
Generate my policy kit →Writing policies for several clients? MSPs, IT consultancies and fractional CISOs keep a roster of client kits that refresh monthly, under their own branding. See partner plans →