Meritos · AI Project Risk Review

Tool version v1.0

AI project risk report

Generated 8 June 2026 · 19 recommendations

This assessment is informed by established AI risk frameworks, including the NIST AI Risk Management Framework and the EU AI Act. It maps to their principles to help structure a first view; it does not certify compliance with any of them.

Summary

"Resolve customer queries automatically — answering product questions, helping with orders, and processing routine returns and refunds — to reduce the volume reaching human agents."

Overview

This looks like a high-risk project.

The driving concern is harm to individuals: you’ve indicated the capability could cause material harm to the people it impacts. The controls that would contain that are not in place: it would be hard to tell when the capability is going wrong, and the capability is able to act with limited oversight. Real exposure without the controls to catch or contain it is what places this in the ‘High risk’ band rather than the middle.

Given you’re still building the capability, the points above are easier and cheaper to address now vs post go-live.

Risk by area

Priority recommendations

These are the key high risk findings and recommended next steps, ordered by assessed criticality:

1. The capability acts on its own, without a real check on what it does
Urgently review the process that the capability follows and the autonomous actions it can take at decision points. Agree a set of explicit, written boundaries covering what the capability may do without a person confirming it first, and treat anything beyond that boundary as needing human sign-off. Ensure that the governance drives action to revisit these boundaries whenever the vendor or the model changes what it is capable of.

2. You might not spot a failure, and the cost of one is not low
Urgently ensure that the process being followed by the capability is documented, including decision points, possible outputs and potential costs of errors / failure at these points. Be clear on what would tell you this capability has gone wrong, and make sure this is being monitored, so a serious problem reaches you through your own checks rather than through the people it affected.

3. The capability can act or reach outside the organisation, and its resistance to manipulation is unestablished
Urgent action is needed to set a hard limit on what the capability can do on its own with content it did not generate, and test how it behaves when fed deliberately hostile input. Both the level of vulnerability and the scope of autonomous actions need to be reviewed to ensure that the capability is operating within safe guardrails.

The decision profile

It is not fully clear who would fix this, or whether they could

Worth noting

You’ve indicated that the knowledge about who would fix this capability, and whether they could, is thinner or less worked-out than it should be. It’s easy to assume that someone will be able to fix it, until this assumption is tested. That risk should be discussed and skills / experience documented to prevent a routine problem turning into a much longer fix.

What to do nextReview skills and knowledge in the support team and make sure more than one person understands this capability well enough to support it, or that your vendor arrangement genuinely covers the things that go wrong with AI rather than just whether the service is running. The aim is that a problem has a clear owner who can actually act on it.

What this could cost to run is not bounded

Worth noting

You’ve indicated there is no firm limit on how high this capability’s running cost could go. AI cost usually scales with use rather than staying fixed, so a spike in demand, a change in how it is used, or a process that runs longer than expected can easily produce a bill that is larger than planned. This is an operational cost risk in its own right that should be documented and managed.

What to do nextPut a ceiling and an alert on what this capability can spend, so an unexpected spike is caught and capped rather than discovered on an invoice. Where it acts autonomously, this limit is the difference between a contained surprise and an open-ended one.

The people picture

The people affected may not know the AI is involved

Important

You’ve indicated that the people this capability affects are not clearly told an AI is part of the decision - either by oversight, or as a deliberate choice. Where a capability affects people, knowing it is involved is what lets them question an outcome, exercise their rights, or simply understand what happened to them. Both UK data protection law and the EU AI Act lean towards disclosure for exactly this reason. An outcome that is delivered without that context can easily be the one that generates a complaint or a challenge that is hard to defend.

What to do nextUrgently review whether to tell affected people about the AI’s role and check whether people have a clear way to ask about AI involvement, including asking for more details about how the AI is used.

There may be no real way to challenge or reverse an outcome

Important

You’ve indicated that the route for an affected person to challenge or reverse an outcome where AI was involved is limited, absent, or not yet thought through. Where an individual has been impacted but has no (clear) recourse can be the difference between correcting a mistake and a much bigger issue involving escalations, compensation and media. In the same way as processes that do not use AI, impacted individuals need to have a way to challenge.

What to do nextMake sure there is a clear, usable way for someone to contest an outcome and reach someone who can discuss and change it. Check that an affected person could actually find and use this process.

The capability affects people who may be vulnerable

Important

You’ve indicated that the people this capability affects include groups who may be vulnerable. Vulnerability raises the importance of every other gap in this report: an unclear outcome, a weak route to challenge it, or a wrong decision all have a higher impact on someone less able to absorb or contest them. It also draws closer scrutiny from regulators, who look first at how a capability treats the people least able to protect themselves. The parts of this capability where an error would fall on a vulnerable person should be held to a higher standard than the rest.

What to do nextIdentify the points where an error would affect a vulnerable person, and ensure that any the process supports clearer communication, a stronger route to challenge, and more human involvement at the point of decision.

The data and technology foundation

The capability can act or reach outside the organisation, and its resistance to manipulation is unestablished

Urgent

You’ve indicated two things that lead to a very high risk when they occur together: the capability acts on its own, connects across systems, or processes content from outside your organisation, and whether it can be manipulated through its inputs has not been established. This is because a manipulated suggestion can be contained when a person reviews it, but a manipulated action the capability takes itself can very often only be undone after the fact rather than contained as part of the process. This is no longer theoretical: in 2025 a state-sponsored group hijacked an AI coding agent to run cyber-attacks largely on its own, and security bodies now report that autonomous agents account for around one in eight reported AI breaches. This very high risk needs to be addressed urgently to create a level of control over an autonomous capability that currently is vulnerable to manipulation.

What to do nextUrgent action is needed to set a hard limit on what the capability can do on its own with content it did not generate, and test how it behaves when fed deliberately hostile input. Both the level of vulnerability and the scope of autonomous actions need to be reviewed to ensure that the capability is operating within safe guardrails.

The legal basis for using personal data here is not pinned down

Important

You’ve indicated the lawful basis for using personal data in this capability has not been fully worked through - relied on in general terms, acknowledged as a gap, or not yet established. Under UK and EU data protection law the basis needs to be settled for the specific use, not inherited from the wider business. This looks at the processing rather than the model and where the specific lawful basis is not clear, regulators have shown they will act on it: the Italian regulator fined OpenAI EUR 15 million in early 2025 over data-protection failings including the basis for using personal data. This legal standing should be confirmed and documented before the capability goes further.

What to do nextConsult with an expert who has a data-protection remit to confirm the lawful basis for this specific use, in writing. It is usually a short piece of work, and far cheaper to settle now than to reconstruct if it is ever questioned.

Where the data came from is not clearly established

Important

You’ve indicated the origin of the data behind this capability is not clearly established - repurposed from another consent, scraped, supplied opaquely by a vendor, or simply unknown. Clear data provenance will need to be shown, documented and proven, if a rights-holder, a regulator, or a customer asks where the data came from. Not being able to trace data is a clear risk - verifying that training and input data was lawfully obtained is increasingly treated as the deploying organisation’s responsibility, not just the vendor’s. Additionally, data that does not have a clear origin is susceptible to errors, either introduced by AI collection or propagated from a data quality issue further up the chain. The data chain should be established and recorded so that it can be assessed and proved if needed.

What to do nextAsk the vendor or data owner for a clear account of where the data originated, how and on what basis it was collected. If this does exist, it should be properly documented and if they cannot give one, this needs to be addressed urgently to avoid potential legal ramifications.

How the capability was tested is informal or leans on someone else

Important

You’ve indicated the capability was tested functionally, informally, on the strength of an upstream supplier, or not yet at all. Functional testing confirms the capability runs; it does not confirm it behaves well on the awkward cases, the edge cases, or the people least like the average. These cases need to be understood and a testing decision made consciously as these are the cases most likely to lead to questions about the credibility and suitability of the AI. What ‘tested enough’ means for this specific use should be defined, the testing against it should be agreed and this decision documented.

What to do nextDecide what ‘tested enough’ means for this specific use, and check whether the testing currently in place actually meets it. The cases worth testing hardest are the ones where a wrong output would have the worst negative impact, not the ones easiest to run.

Once live, capability monitoring is thin

Important

You’ve indicated that monitoring in production is limited to functional health, vendor dashboards, raw logs, or nothing in particular. A capability can run perfectly at a technical level while its outputs drift, degrade, or start landing badly on the people they affect - and functional monitoring will not show you that. Uptime monitoring is easily confused with quality monitoring / observability: the service is ‘up’ while its judgement and outcomes degrade unnoticed. Monitoring needs to include observability: making sure it is clear what the capability is doing and ensuring this is still working as expected and producing reliable, defensible outputs. This requirement should be defined and implemented now.

What to do nextAgree how the capability will be observed for quality, not just availability and what signals would tell you this capability has started going wrong, and make sure something is actually watching them. Vendor dashboards are evolving to cover this need but do not always cover this; the entire system needs to report judgement as well as uptime.

How data quality is handled is informal or outsourced

Worth noting

You’ve indicated that data quality is handled informally, relied on from upstream, or not yet addressed. A capability is only as reliable as the data feeding it, and quality problems tend to show up as confident, wrong outputs rather than obvious errors - which makes them easy to miss until they have had an effect. This is one of the most common reasons AI projects underperform in practice: the issue is rarely the model, and far more often the data feeding it. Ownership of data quality for this capability should be defined rather than assumed and the relevant standards agreed so that data quality can be shown to be managed and intentional.

What to do nextConfirm who owns data quality for this capability and what ‘good enough’ looks like in practice. Once this is clear, establish a process to manage data quality and, if necessary, conduct a data quality audit to set a baseline. This can be upstream but needs to be visible and aligned with the outcome expectations.

You cannot fully trace the data the capability used

Worth noting

You’ve indicated that the data flow behind this capability - which data fed which output, and where it came from at each step - is only partly visible or not yet mapped. Lineage is what lets you answer ‘why did it do that’ after the fact. Without it a regulator’s or customer’s question can be hard to answer with confidence because tracing what decisions were made on is mostly guesswork. This gap is often noticed only when an output is challenged when it’s a huge effort to reconstruct the decision path and data relied on. It is far easier to build lineage in now than to reconstruct it later.

What to do nextTake a representative sample of recent outputs and check whether you could trace the data behind them. If you can’t this should be addressed as soon as possible to avoid having to unpick things later.

The capability rests on a vendor or product that is still maturing

Worth noting

You’ve indicated the capability comes from an emerging supplier, a new or evolving product, or one whose maturity you are not sure of. A maturing capability is not a problem in itself, but it carries more risk of change and less proven track record than one later in its lifecycle: the product can change quickly, support can be thinner, and there is less history to tell you how it behaves over time. This dependency should be managed rather than assumed stable, with a clear view of what you would do if the product or the supplier changed.

What to do nextDiscuss the different scenarios for product or vendor change and agree what actions would be taken if this vendor changed the product materially, raised the price, or went away. A clear answer now is easier and quicker to deploy than an improvised one later.

The contract may not cover how the capability is actually used

Worth noting

You’ve indicated the contractual position rests on standard terms, terms you cannot fully see, or terms the supplier dictates. Standard software terms often predate the way AI capabilities are used, and can be unclear on the questions that matter here: who is accountable for an output, what the supplier may do with your data, and what happens if the capability causes harm. This can be true for both pure-play AI vendors as we as multi-product providers as it’s easy to rely on old templates. The risk here is that the contract terms are not clear, or absent entirely, about key elements such as liability, data use and AI accountability. These should be clear before they are needed.

What to do nextEnsure that a full review (ideally by a legal expert) has been done on the relevant contracts to check whether they address liability, data use, and accountability for the AI specifically. If these are missing then they should be discussed to agree relevant terms with the vendor/s and the contracts updated.

The governance picture

The capability acts on its own, without a real check on what it does

Urgent

You’ve indicated the capability does more than suggest or inform - it takes actions - and the human check around those actions is limited or inconsistent. This is very high risk as allowing the AI capability to act without a reliable check is the combination most likely to produce a consequence that is both unexpected and has a negative impact on either the user or the organisation. The more actions the capability can take without being asked, the more this matters, and the recent rise in incidents involving autonomous agents has come largely from capabilities with this risk. An explicit boundary on what the AI capability may do unsupervised should be set.

What to do nextUrgently review the process that the capability follows and the autonomous actions it can take at decision points. Agree a set of explicit, written boundaries covering what the capability may do without a person confirming it first, and treat anything beyond that boundary as needing human sign-off. Ensure that the governance drives action to revisit these boundaries whenever the vendor or the model changes what it is capable of.

You might not spot a failure, and the cost of one is not low

Urgent

You’ve indicated two things that create a higher risk together than individually: you would not reliably detect when the capability has gone wrong, and the cost if it does is not low - material or serious on your answers, or not yet established. This risk is that a failure you have no visibility of continues until it is reported to you by an external who has seen an impact, likely a customer or the regulator. This often leads to high impact whatever the actual cost but you have indicated that the cost of failure is not low so this failure could have a big negative impact on cost, reputation or both. A specific detection check for the most costly failure should be put in place.

What to do nextUrgently ensure that the process being followed by the capability is documented, including decision points, possible outputs and potential costs of errors / failure at these points. Be clear on what would tell you this capability has gone wrong, and make sure this is being monitored, so a serious problem reaches you through your own checks rather than through the people it affected.

No one is clearly accountable for this capability

Important

You’ve indicated accountability for the capability is collective, unclear, or unassigned. The risk is that shared accountability can become no accountability: when something goes wrong, a capability that was ‘everyone’s’ turns out to be no one’s, and the response stalls while ownership is worked out. The ultimate owner for the capability should be agreed and documented as soon as possible, ideally at both operational and executive level. This will support decision making as well as the right focus for handling problems and improvement opportunities.

What to do nextDiscuss and agree accountable owner/s for this capability, including their consent. Document these and communicate to any individuals / teams who might be involved in incidents or improvements.

Regulatory and reputational lens

How this would look from the outside is unsettled

Worth noting

You’ve indicated some unease about how this project would look if it drew outside attention, or that it has not yet been considered through that lens. Reputational exposure can come from the capability itself but is often from the gap between how it works and how it would be described once something has gone wrong. The risk here is that the potential exposures may be unknown and hence hard to quantify risk levels against.

What to do nextConfirm that the process, decision points and actions allowed for the AI are detailed for this capability. Within these, identify the aspects that are likely to have the highest interest from a journalist or regulator. If these elements have the weakest controls then this needs to be strengthened to both make the process more robust and make it more defensive to external review.

This capability is likely within scope of the EU AI Act

Informational

You’ve indicated the capability reaches people or operations in the EU. The EU AI Act applies on the basis of where the people affected are, not where the organisation are based, so a capability run from the UK or elsewhere can still fall within it. This is not a finding about anything being wrong - it is a note that the EU framework is one of the lenses this project sits under, and the obligations scale with how the use is classified rather than applying uniformly. Where this use sits in the Act’s risk tiers is worth establishing rather than assuming.

What to do nextEstablish where this use sits in the Act’s risk tiers, since that determines what, if anything, is required.

About this report. This is a structured reading of your AI project based on the answers you provided. It is not a compliance assessment, regulatory verdict, or substitute for a DPIA. Where it flags concerns, those are starting points for conversations within your organisation, not conclusions.

If it would help to talk it through. If you would like to discuss any of this with our expert team, Meritos is reachable at hello@meritosconsulting.com.

Methodology. The tool runs entirely in your browser. No answers, identifiers, or report content leave your device. Two anonymous counts are kept; that a review was started, and that a report was generated - nothing else. The framework draws on the EU AI Act, UK ICO guidance, NIST AI RMF, and the MIT AI Risk Repository, with Meritos's own analytical view of how these compose for enterprise AI projects. Produced with Meritos AI Project Risk Review v1.0.