Building Generative AI Into a Healthcare Product

Generative AI has a clear role in healthcare products: it can draft clinical documentation for human review, summarize patient records and messages, and answer administrative questions that do not involve diagnosis. What it should not do is provide a diagnosis or treatment recommendation without review by a licensed clinician. That crosses into regulated medical-device territory.

This distinction, drafting and summarizing versus diagnosing and recommending, shapes most of the decisions that follow. It determines what the model can say, what requires human review before reaching a patient, and whether the feature may need FDA review.

Where Generative AI Fits in a Healthcare Product

Three product categories account for most of the generative AI features shipping in healthcare products today.

• Ambient clinical documentation: listens to or reads a visit and drafts a note in the provider's format, which the provider edits and signs.
• Patient message and record summarization: condenses a long chart or message thread into a short summary a staff member reviews before acting on it.
• Administrative assistance: answers scheduling, billing, and benefits questions that do not touch diagnosis or treatment, and hands off to a human for anything that does.

Where It Does Not Belong Without a Human in the Loop

A feature that suggests a diagnosis, recommends a treatment, or triages a patient's symptoms without a licensed clinician reviewing the output before it reaches the patient is functioning as clinical decision support, and the FDA regulates that category as a medical device in many cases.

The fix is not avoiding the feature. It is keeping a clinician in the review loop before any AI-generated clinical judgment reaches a patient, and documenting that review step, since the review step itself is usually what keeps a feature out of the highest-risk device classification.

HIPAA and Where the Model Runs

Any generative AI feature that processes patient data needs a signed Business Associate Agreement (BAA) with the model provider before that data reaches the model, the same requirement that applies to any other vendor touching protected health information (PHI).

Most major model providers now offer a BAA-covered enterprise tier for exactly this reason, but the default consumer-facing version of the same model typically does not carry one. Confirm which tier a team is using before assuming a feature is compliant. Across the SMB workflows we automate, the most common compliance miss is a developer testing against a personal, non-BAA account and forgetting to flip the project to the enterprise tier before launch.

• Confirm a signed BAA exists before any patient data reaches the model, in testing and in production alike.
• Confirm data retention and training-use settings match a healthcare use case, since a default consumer setting may retain data longer than a compliance team expects.
• Log every AI-generated output that a clinician reviews and edits, since that log is often what an auditor asks for first.

What Building This Feature Runs

Adding a single well-scoped generative AI feature, ambient documentation for one visit type, or message summarization for one inbox, typically costs 15,000 to 40,000 dollars to build for a small to mid-size healthcare product, including the compliance and review-workflow work, not just the model integration.

Ongoing model costs run separately, usually a few hundred to a few thousand dollars a month depending on visit or message volume. The compliance and audit-logging work is usually the larger and more variable part of the budget, not the model API cost itself, and teams that scope only the model integration tend to underestimate the total build by a wide margin.

The review-workflow piece specifically covers three things: the interface a provider uses to accept or edit an AI-drafted note, the audit log that records every edit, and the testing process that checks output quality after each model version change. Skipping any one of the three is the most common way a project comes in under budget on paper and over budget once compliance review starts.

Building a Custom Model Layer vs. Integrating an Existing Tool

A healthcare product adding its first generative AI feature is almost always better served integrating an existing model through a BAA-covered API than training or fine-tuning a custom model, since the compliance groundwork alone takes months to redo from scratch.

A custom fine-tuned model earns its cost once a product has enough of its own clinical documentation to train against and a specific format or specialty that a general model handles poorly out of the box. Most products never reach that point.

Fine-tuning also raises a separate compliance question. The training data itself is patient documentation, so the same BAA and audit-logging requirements apply to the training pipeline, not just the finished model. A team that fine-tunes without extending its compliance review to the training step has only solved half the problem.

• First generative AI feature, general documentation or summarization: integrate a BAA-covered model through an API.
• High volume, one narrow specialty, general model output needs heavy editing: fine-tuning may be worth the added cost.
• Either path still needs the human-review workflow and audit logging described above, regardless of which model choice you make.

Failure Modes That Show Up After Launch

Three problems account for most generative AI healthcare features that get pulled after launch, and each one is avoidable with the right review step designed in from the start.

• Silent hallucination in a clinical note: a drafted note states something the visit did not actually cover. Fix it by requiring the provider to actively confirm each AI-drafted section, not just click one blanket approval button.
• Model drift after a provider update: an upstream model update changes output tone or format without warning. Fix it by testing against a fixed set of sample visits after every model version change, before rolling the update out to real users.
• Review fatigue: a provider stops carefully reading AI drafts once volume gets high, defeating the human-in-the-loop safeguard. Fix it by keeping the AI-drafted sections visually distinct from the provider's own edits, so a skimmed review still catches an obvious error.

Who This Approach Is Not For

A product still validating its core clinical workflow with real users should not add a generative AI feature yet. The reason is simple. A human needs to understand the documentation or summarization problem first, or the AI feature just automates an unproven process faster.

The calculation changes once the underlying workflow is stable and repeated often enough that a provider's time on it is a real, measurable cost, and once the compliance team has bandwidth to build the review and audit-logging workflow properly, rather than bolting it on after a rushed launch.