A 70-person digital agency got its project managers out of the support queue with an AI first line and a human approver

The problem

On a Tuesday in February, the senior project manager on the agency’s largest account blocked two hours for sprint planning. She got eleven minutes into it.

The client, a personal finance education company, talked to the agency in three shared Slack channels. By noon, those channels carried 31 messages. Eleven were thank-yous, nine were how-to questions she had answered before, and one was a real bug.

The bug sat under the noise for three hours and twenty minutes. The client’s loan repayment calculator was showing the wrong monthly payment to its users. By the time she found it, the client’s head of product had posted, “Is anyone watching this channel?”

Sprint planning moved to Thursday. The release slipped a week, and the $48,000 milestone invoice tied to it slipped with it. She was the best PM the agency had, and she had become the most expensive help desk in the building.

Where Phos AI Labs came in

Phos AI Labs started with a four-week AI Readiness Audit. We interviewed six project managers, the head of delivery, both tech leads, and the founder. We sat in on two client standups.

Then we read the messages. We logged and labeled 600 client messages from the largest account’s three channels. We also reviewed 90 days of the founder’s business WhatsApp and the texts PMs were sending clients from their personal phones.

Three findings shaped the work. Thirty-eight percent of messages were how-to questions, most already answered somewhere in Slack history. Another 22% were pure noise. And there was no written knowledge base, so every answer lived in one PM’s head.

The audit also surfaced two quieter gaps. Clients who preferred texting reached PMs on personal phones, where nobody else could see the thread. Business WhatsApp chats sometimes sat unanswered for three days.

The baseline

Before we proposed anything, we put numbers on the load. The numbers below became the scoreboard for the engagement.

The 13 hours stood out. A third of the PM’s week went to answering messages a well-kept knowledge base could have answered. The hours came straight out of planning and delivery.

Setting the foundation

The AI Foundation phase took two weeks. We sat with the head of delivery and the senior PM and wrote down how support should work, because nobody had written it down before.

First came the categories. Every client message is a bug, a formula question, an enhancement, a feature request, a how-to, or noise. Each category got a definition, five real examples from the audit, and an owner: bugs and formulas go to a tech lead, enhancements and features go to the PM for scoping.

Then the knowledge base format. Each entry holds the question, the approved answer, the source, the date, and who approved it. The senior PM seeded it with nine answers she gave every week.

We wrote the approval policy into the same document. No reply reaches a client until a named approver says yes. Database access is read-only, and no draft may promise a date, a price, or a new feature.

Deciding what to build

With the baseline and the playbook in hand, the Phos AI Labs engineering team joined the account. Their job was to decide what kind of system each problem needed.

We looked at three common options first. A help desk tool like Zendesk would ask clients to leave Slack, and they had said no. A chatbot answering clients directly could quote a wrong loan formula to a finance company’s users. A support coordinator hire would cost about $60,000 a year and still start with an empty knowledge base.

So we matched the tool to the job. Repeatable steps with no judgment became automations: the client text bridge and the WhatsApp watch. The noise check became an AI-enhanced workflow, one cheap decision placed at the front of an existing process.

The AI employee was the first line itself. The support AI employee owns first-line support for the account: it reads every message, sorts it, checks the facts, drafts the answer in the agency’s voice, and learns from every correction its approver makes.

The approver is its manager. The senior PM took the role, because reviewing a finished draft takes a minute and answering from scratch took ten.

The build

The build ran five weeks. The support AI employee checks the three client channels every 60 seconds, using the agency’s existing Slack bot. Every new message goes through four stages, each run by the model best suited to it.

Stage one is the noise gate. A small decision model answers one question: is this pure noise? It costs about two thousandths of a cent per message. Anything scoring above 0.85 is dropped and logged, never deleted.

Stage two is the classifier. A fast model assigns one of the six categories and a confidence score. Below 0.7 confidence, the message goes to the approver unlabeled, with the two likeliest categories shown side by side.

Stage three is the check. For bugs and formula questions, a stronger reasoning model reads the knowledge base and the matching records in the client’s Supabase database, through a read-only key. It writes down what it found and what it could not confirm.

Stage four is the draft. A drafting model writes the reply in the agency’s voice, citing the knowledge base entry or the record it used. Enhancements and feature requests get a short acknowledgment draft plus a scoping note for the PM.

Each draft lands in the approver’s Slack DMs with a short ID. She replies “yes 14”, “no 14”, or “edit 14” followed by her version. Yes posts the reply in the client’s thread. Every edit is saved to the knowledge base, so the next similar question starts from her words.

Running the whole stack costs about $85 a month. Because each stage is separate, we can swap any single model when a better one ships, without touching the others.

The client text bridge runs on a Twilio number. A teammate posts in a Slack channel to text a client, and each phone number gets its own thread, with replies landing back in it. The WhatsApp watch logs every 1:1 business chat and, every three hours, flags any message left unanswered for 24 hours. Group chats are skipped, since the team reads those live.

Rollout and training

Clients never saw the first version. For two weeks the system ran in shadow mode. It drafted replies no client saw, and the senior PM compared them against what she actually sent.

The comparison was the first round of training. She saw where drafts were right, flagged where they guessed, and got used to the yes, no, and edit replies before anything counted.

Go-live covered the largest account’s three channels in week six. The text bridge and the WhatsApp watch went live the same week for the whole agency.

Training ran in three sessions. A 90-minute session taught all six PMs to route support to the first line instead of answering in the channel. The approver session covered writing edits that teach well. The tech leads learned how bug and formula drafts reach them.

Ownership is written down. The senior PM approves drafts. A tech lead owns the database check rules. The head of delivery owns the knowledge base and a 20-minute weekly review, so changes no longer wait on Phos AI Labs.

Making it better

The early drafts were thin. The approver edited 71% of drafts. With only nine knowledge base entries, the drafting model filled gaps with polite guesses, and she rewrote most of them.

The noise gate made one real mistake. In week one it dropped “thanks, also the export is broken again” as a thank-you. The log caught it the same day. We changed the rule so any message with a question or a second clause always passes the gate.

The classifier confused formula questions with bugs about a third of the time. We added twelve labeled examples of each to the playbook, and the confusion fell below one in ten within two weeks.

In month two, a newer small classifier model came out. We tested it against all 600 labeled audit messages before switching. It sorted 94% correctly, up from 88%, so we swapped it in.

The knowledge base did the rest. Every edit became an entry, and by week twelve it held 136 approved answers. The share of drafts approved with no edit rose from 29% to 58%.

The outcome

On a Tuesday in June, a client user reported that the repayment calculator showed the wrong payment for one plan. The support AI employee classified it as a formula question within a minute.

The read-only check found the interest rate for that plan stored as 68 instead of 6.8. The draft named the record, and the tech lead got the fix request with it. The senior PM approved the reply with “yes 212” from sprint planning, which ran on time.

Fourteen thank-yous came through the channels that day. None of them reached her. The fix shipped that afternoon.

The nine hours a week went back into project work. The classifier logged 41 distinct enhancement and feature requests in the quarter, each with a scoping note, and seven became signed change orders worth $86,000.

What’s next

The project doesn’t have an end date. Each edit adds to the knowledge base, and each new model generation sorts and drafts more accurately. We retest every model release against the same 600 labeled messages before switching.

Next, the support AI employee extends to two more client accounts, each with its own knowledge base. The target is 75% of drafts approved with no edit, and then lighter approval for the question types that have proven safe.

If your project managers spend their days answering support, start with an audit. Phos AI Labs will tell you how much of their week comes back.

Client details changed under NDA.