Where the language model should stop: Designing AI for payroll, admissions, and other areas that punish errors – Unite.AI

I’ve spent the last few years integrating AI into products where a wrong answer comes with a price: an AI-powered salary compliance platform for the Australian market, where one mistake ends up in someone’s bank account, and a college admissions product, where a poor match can impact a student’s next four years. The question that decided the architecture was the same in both. How far should the language model go before something deterministic takes over?

One in five payrolls in the United States contain an error, costing an average of $291 each before litigation comes into play. The EU AI law places AI used for admission decisions and employment management in its high-risk categories.

Let the rules dominate the numbers

Deterministic rules own anything that has a dollar figure or legal consequence attached to it, and the model interprets and explains around those rules, and nothing more.

On the payroll platform, our goal was a clear boundary between payroll rules and AI. Payroll experts first define the salary rules in a structured format. The system then processes these rules and generates every possible scenario that can arise from them. This complete set of scenarios becomes the model’s source of truth. The model cannot create new rules, modify existing ones, or decide which sources to use. It can only retrieve the relevant ruleset, look up rate tables, and cite the scenario used. Rule creation, scenario generation and source selection remain completely outside the AI ​​layer.

The admissions product follows the same logic: location requirements, budget limits, institution status and eligibility thresholds are applied as rules before the model can influence a ranking. If a student says they need to attend college in California, an out-of-state school can’t ignore that because the model thought it would otherwise be a good fit. The model handles softer preferences and explanations, and deterministic logic acts as a barrier where approximately right is not acceptable.

Recovery grounds the model, up to a point

Augmented recovery generation keeps responses tied to organizational policies rather than whatever the model absorbed in training. It is also oversold. When Stanford’s RegLab tested RAG-based commercial legal research tools, they still hallucinated between 17% and 33% of the time. Recovery restricts what the model can say, and the model can still be wrong.

I learned it first hand. On the payroll platform, the system once found the correct work premium for a case, but the model still interpreted the overtime rules incorrectly. The right source had been retrieved, but the answer was wrong, so we later became much more strict about what the model could decide on its own. Important payroll rules were moved to explicit business logic, every answer had to be supported by source material, and we added unsuccessful case-based tests. Access to the right information only helps if you also control how the model uses it.

Make every response traceable

Before a response is generated, the system records exactly which regulation and version was used, along with the template and other details of the request. To say that an AI response was based on company policy is difficult to verify. Saying that it was generated using version 7 of a specific agreement, which has been preserved and can be compared with other versions, is much stronger evidence. We also track each model and tool call: what the user asked, what was retrieved, what tools were run, what the model returned, and any errors along the way. When a support issue arrives, we can see exactly where an incorrect response is coming from. The 2024 Air Canada ruling, in which a court forced the airline to grant a discount invented by its chatbot, shows what the alternative costs are.

Put humans where the consequences are

NIST’s Generative AI Profile addresses the line between what a system does on its own and what a person requires as a governance decision. We escalate a case to a human when the system is uncertain, when information is conflicting, or when the decision could significantly affect the user. In admissions, an unusual profile of a student, or a recommendation that clearly does not meet a student’s strict requirements, is flagged for advisor review. This step caught recommendations that seemed reasonable on the surface but didn’t meet a constraint in the student profile, and each one was fed back into the system. Neither product finalizes anything on its own. Consultant intervention is required before a recommendation is relied upon, and each payslip generated by the payroll management system is flagged for approval before being published. Human review is not designed for every request. The goal is to confidently automate routine cases and engage a person when the system has a good reason to be uncertain.

Decide what the model can possess today

There is a practical test that I apply to any workflow. If a bad output can be detected and canceled before it reaches a person, the model can own that step. If the answer can be checked against a rule, the rule has it and the model explains it. If a bad result were to reach a paycheck or application decision before anyone sees it, that step isn’t ready.

Language models are the best interpreters we have ever put in front of a regulation. In fields that punish mistakes, interpretation is where they should stop.