AI Governance Documentation — What Your Program Actually Needs

Most AI governance programs produce the wrong documents.

They start with a policy statement, hold a kickoff meeting, and generate a slide deck summarizing the governance vision. Then the slide deck sits in a shared drive and nothing about how AI is actually used at the company changes.

Effective AI governance documentation is not about vision. It is about accountability: knowing what AI tools you have, who owns them, what they are allowed to do, and what happened when something went wrong.

This guide covers the specific documents a working AI governance program requires, who should own each one, and how to keep them current.

The documents that actually matter

An AI governance program at a small to mid-market organization needs six core documents. Everything else is optional.

1. AI policy
2. AI inventory
3. Model cards (or tool cards)
4. Risk assessments
5. AI incident log
6. Audit record

Each is explained below with what it should contain, who owns it, and how often it needs updating.

For the broader framework that these documents sit inside, see building an AI governance framework. For a complete template you can copy and adapt, see the AI governance framework template.

Document 1: AI policy

What it is: The foundational document that defines how your organization evaluates, approves, and monitors AI use.

What it must contain:

• Scope (which tools, which people, which use cases it covers)
• Risk tier definitions (low / medium / high, with examples)
• Approval process for new AI tools
• Prohibited uses
• Employee responsibilities
• Incident reporting requirements
• Review schedule

What it does not need to contain: Technology architecture, vendor lists, implementation details, or aspirational statements about AI strategy. Those belong in separate documents.

Who owns it: The AI Steering Committee or whoever holds AI governance accountability at your organization — typically the COO, general counsel, or a designated AI lead.

How often it updates: At minimum, annually. In practice, review it when you adopt a materially new category of AI use (e.g., moving from productivity tools to AI in customer-facing decisions) or when relevant regulations change.

Common mistake: Writing a policy that describes what AI can do rather than how decisions about AI get made. The policy should be a process document, not a technology document.

Document 2: AI inventory

What it is: A live register of every AI tool your organization has approved for use.

What it must contain (per tool):

• Tool name and vendor
• Version or product tier in use
• Approved use case(s)
• Risk tier classification
• Data types the tool accesses (customer, employee, financial, proprietary)
• Named AI owner (the employee accountable for the tool)
• Approval date and approving party
• Next scheduled review date
• Current status (active / under review / decommissioned)

Who owns it: A designated document owner — typically the IT lead or operations lead — with AI owners responsible for keeping their individual entries current.

How often it updates: Continuously. Every new tool approval adds a record. Every decommissioned tool gets a status change. Review the full inventory at minimum quarterly.

Common mistake: Building the inventory only for new tools approved after the governance program launches. Before you do anything else, audit every AI tool already in use at your organization. Most companies find 10–30 tools in active use with no documentation. Start there.

Format: A well-maintained spreadsheet works for organizations with fewer than 25 active tools. For larger inventories or organizations with compliance requirements, a GRC platform or purpose-built AI registry may be appropriate.

Document 3: Model cards (tool cards)

What they are: One-page documentation records for each approved AI tool, capturing its capabilities, limitations, and approved operating parameters.

Model cards originated in the machine learning research community to document trained models. For most organizations, you will not be documenting internally built models — you will be documenting commercial AI tools. The same structure applies.

What each card must contain:

Identity

• Tool name, vendor, version
• Date of current card
• AI owner

Intended use

• Approved use cases (specific and bounded)
• Approved user groups
• Approved data inputs

Known limitations

• What the tool does not do well
• Error types or failure modes observed
• Conditions where outputs require extra scrutiny

Data handling

• What data the tool retains
• Where data is processed and stored
• Data deletion or opt-out options

Monitoring notes

• How outputs are reviewed
• Who reviews them and how often
• What triggers an escalation

Who owns each card: The AI owner for that tool.

How often they update: When the tool’s approved use changes, when the vendor releases a material update, or when monitoring reveals new limitations. At minimum, review annually.

Common mistake: Skipping model cards for “low-risk” tools. The discipline of documenting limitations is most valuable before a problem occurs, not after.

Document 4: Risk assessments

What they are: Pre-deployment evaluations that establish how a proposed AI tool was evaluated before approval.

When they are required: For any tool classified as Tier 2 (medium risk) or Tier 3 (high risk). Tier 1 tools may use a simplified checklist rather than a full assessment.

What a Tier 2 / Tier 3 risk assessment must contain:

Use case description

• What the tool will do, specifically
• Which workflows it affects
• Who makes decisions based on its outputs

Risk identification

• What could go wrong?
• What is the potential harm to customers, employees, or the organization?
• What is the likelihood and severity of each identified risk?

Regulatory assessment

• Does this use case implicate GDPR, CCPA, HIPAA, FCRA, ECOA, or other regulations?
• What obligations does the organization have under each?

Vendor assessment

• Security certifications held
• Data retention and deletion terms
• Incident notification commitments
• Subprocessor list reviewed

Mitigations

• What controls reduce identified risks?
• What human review process is in place?
• What are the conditions under which the tool would be suspended?

Approval decision

• Approved / Denied / Approved with conditions
• Approving party and date
• Conditions, if any

Who owns each assessment: The AI owner, with review by legal/compliance for Tier 3.

How often they update: When the use case changes materially, when a vendor changes their data terms, or when monitoring reveals risks not identified in the original assessment.

Document 5: AI incident log

What it is: A running record of every AI incident — cases where an AI tool produced outputs that caused harm, violated policy, or operated outside its approved parameters.

What each entry must contain:

• Incident ID and date
• Tool involved
• Who identified the incident
• Severity classification (low / medium / high)
• Description of what happened
• Immediate action taken (tool suspended? outputs recalled?)
• Root cause determination
• Remediation steps
• Who completed remediation and when
• Whether the framework or policy was updated as a result

Who owns it: The AI Steering Committee or designated governance lead, with AI owners contributing entries for their tools.

How often it updates: When incidents occur. Review the full log quarterly to identify patterns.

Why it matters beyond compliance: The incident log is your organization’s institutional memory about where AI fails. Reviewing it regularly reveals whether specific tools have recurring issues, whether certain use cases are inherently higher risk than classified, and whether your monitoring processes are catching problems early enough.

Common mistake: Only logging incidents that caused external harm. Log near-misses, internal quality failures, and cases where outputs were caught and corrected before any harm occurred. Those cases tell you whether your review processes are working.

Document 6: Audit record

What it is: Documentation of periodic reviews confirming that your governance program is functioning as designed.

What it must contain:

• Audit date and period covered
• Auditor(s)
• Scope (which documents, which tools, which processes were reviewed)
• Findings (what was in compliance, what was not)
• Gaps identified
• Remediation actions assigned
• Sign-off

Who owns it: The AI Steering Committee chair or equivalent.

How often it updates: Annually at minimum. More frequently for organizations with Tier 3 tools or regulatory obligations.

What the audit should cover:

• Is the AI inventory complete and current?
• Do all active tools have current model cards?
• Are Tier 2 and Tier 3 tools being monitored at the required frequency?
• Is the incident log being maintained?
• Have risk assessments been updated when required?
• Have any tools been used outside their approved use cases?
• Is the policy itself still accurate given current AI use?

Documents you probably do not need yet

Many AI governance frameworks recommend documents that add administrative weight without proportional value for small to mid-market organizations.

Algorithmic impact assessments — Valuable for organizations building or fine-tuning their own models. Most SMBs are using commercial tools and do not need this level of documentation until they are developing proprietary AI systems.

Bias testing reports — Required for high-risk use cases in regulated industries (hiring, lending, insurance underwriting). Not necessary for most productivity, operations, or content use cases.

Data lineage documentation — Relevant for organizations training custom models or managing complex data pipelines. Commercial AI tool users typically do not need this.

Ethics review boards — Appropriate for enterprises or organizations operating at the frontier of AI capability. For most mid-market organizations, integrating ethics considerations into the existing risk assessment process is sufficient.

Build what you need. Add complexity as your AI use and risk profile grows.

Who should own AI governance documentation

The failure mode in most organizations is diffuse ownership — everyone is responsible, so no one is responsible.

For a small to mid-market organization, assign documentation ownership as follows:

Single ownership per document. Named individuals, not departments.

How to start if you have no documentation today

Most organizations starting an AI governance program have no documentation and significant existing AI use. Here is a practical sequence:

Week 1–2: Inventory what you have Ask every team to list the AI tools they use. Include free tools, browser extensions, embedded AI features in existing software, and personal accounts used for work purposes. Do not judge; just collect.

Week 3–4: Classify and prioritize Apply your risk tiers to the inventory. Identify any Tier 3 uses that require immediate attention. Flag tools with unknown data handling terms.

Month 2: Write the policy Document what your governance process will be going forward. Keep it short. Two to four pages is enough for most organizations at this stage.

Month 2–3: Complete model cards for Tier 2 and Tier 3 tools Start with your highest-risk tools. Tier 1 tools can be documented on a rolling basis.

Month 3: Complete risk assessments for any Tier 3 tools already in use These are retroactive. If the tool cannot pass a reasonable risk assessment, that is a decision to make now rather than after an incident.

Ongoing: Maintain the log, run annual audits Once the foundation exists, governance is maintenance: updating records when things change, logging incidents when they occur, and auditing annually.

For a complete governance framework template that incorporates all six document types, see the AI governance framework template. For the workflow that governs how new AI tools move through the approval process, see AI governance workflow.

If your organization needs help designing and implementing an AI governance program that fits your size and industry, Phos AI Labs works with SMBs and mid-market companies on governance alongside full AI implementation.

Further reading

• AI Governance Framework Template
• Building an AI Governance Framework
• AI Governance Best Practices