AI project risk report
This assessment is informed by established AI risk frameworks, including the NIST AI Risk Management Framework and the EU AI Act. It maps to their principles to help structure a first view; it does not certify compliance with any of them.
Summary
Overview
This looks like a high-risk project.
The driving concern is harm to individuals: you’ve indicated the capability could cause material harm to the people it impacts. The controls that would contain that are not in place: it would be hard to tell when the capability is going wrong, and the capability is able to act with limited oversight. Real exposure without the controls to catch or contain it is what places this in the ‘High risk’ band rather than the middle.
Given you’re still building the capability, the points above are easier and cheaper to address now vs post go-live.
Risk by area
What this points to
Pause and redesign
The risk picture here is serious, and the capability is already building or running.
The exposures identified are significant and sit in the design rather than around it, so they are unlikely to be resolved by adding process to what already exists. The uncomfortable part of this reading is that the capability is live or nearly live while that is true, which is what makes it a pause rather than a plan.
Who else to involve
Based on what was raised, these are the conversations worth having, and why:
Legal or data protection
the legal basis for using personal data has not been pinned down; retention of the data and outputs has not been settled; people may not be told AI is involved, and the deployment has EU exposure.
Commercial or procurement
what the contract covers has not been established; the provider is early-stage or unproven for this use; no one is clearly accountable for what the capability costs to run.
Information security
the capability acts on its own or reaches beyond the organisation, and how well it resists manipulation has not been established.
Executive or board
the overall risk reading is high; the capability could cause material harm to individuals.
This assessment sees risk, not value. It cannot weigh what the capability is worth to you, so the posture above is a prompt for your decision rather than a substitute for it.
Priority recommendations
These are the key high risk findings and recommended next steps, ordered by assessed criticality:
1. The capability acts on its own, without a real check on what it does
Urgently review the process that the capability follows and the autonomous actions it can take at decision points. Agree a set of explicit, written boundaries covering what the capability may do without a person confirming it first, and treat anything beyond that boundary as needing human sign-off. Ensure that the governance drives action to revisit these boundaries whenever the vendor or the model changes what it is capable of.
2. You might not spot a failure, and the cost of one is not low
Urgently ensure that the process being followed by the capability is documented, including decision points, possible outputs and potential costs of errors / failure at these points. Be clear on what would tell you this capability has gone wrong, and make sure this is being monitored, so a serious problem reaches you through your own checks rather than through the people it affected.
3. The capability can act or reach outside the organisation, and its resistance to manipulation is unestablished
Urgent action is needed to set a hard limit on what the capability can do on its own with content it did not generate, and test how it behaves when fed deliberately hostile input. Both the level of vulnerability and the scope of autonomous actions need to be reviewed to ensure that the capability is operating within safe guardrails.
The decision profile
It is not fully clear who would fix this, or whether they could
Worth notingYou’ve indicated that the knowledge about who would fix this capability, and whether they could, is thinner or less worked-out than it should be. It’s easy to assume that someone will be able to fix it, until this assumption is tested. That risk should be discussed and skills / experience documented to prevent a routine problem turning into a much longer fix.
What this could cost to run is not bounded
Worth notingYou’ve indicated there is no firm limit on how high this capability’s running cost could go. AI cost usually scales with use rather than staying fixed, so a spike in demand, a change in how it is used, or a process that runs longer than expected can easily produce a bill that is larger than planned. This is an operational cost risk in its own right that should be documented and managed.
This pilot has no clear test to conclude against
Worth notingYou’ve indicated this is a pilot or trial without success criteria and / or stop conditions agreed. The risk is that the pilot cannot really fail: with nothing set beforehand to measure against, an experiment is often judged a success after the fact and carried into production regardless of how it actually performed. A set of metrics and success criteria - what good looks like, and what would make you walk away – should be considered to help ensure that the pilot is meaningful and defensible.
The people picture
The people affected may not know the AI is involved
ImportantYou’ve indicated that the people this capability affects are not clearly told an AI is part of the decision - either by oversight, or as a deliberate choice. Where a capability affects people, knowing it is involved is what lets them question an outcome, exercise their rights, or simply understand what happened to them. Both UK data protection law and the EU AI Act require disclosure for exactly this reason, and since August 2026 the EU AI Act transparency obligations apply directly to organisations deploying these capabilities, not only to those building them. An outcome that is delivered without that context can easily be the one that generates a complaint or a challenge that is hard to defend.
There may be no real way to challenge or reverse an outcome
ImportantYou’ve indicated that the route for an affected person to challenge or reverse an outcome where AI was involved is limited, absent, or not yet thought through. Where an individual has been impacted but has no (clear) recourse can be the difference between correcting a mistake and a much bigger issue involving escalations, compensation and media. In the same way as processes that do not use AI, impacted individuals need to have a way to challenge.
The capability affects people who may be vulnerable
ImportantYou’ve indicated that the people this capability affects include groups who may be vulnerable. Vulnerability raises the importance of every other gap in this report: an unclear outcome, a weak route to challenge it, or a wrong decision all have a higher impact on someone less able to absorb or contest them. It also draws closer scrutiny from regulators, who look first at how a capability treats the people least able to protect themselves. The parts of this capability where an error would fall on a vulnerable person should be held to a higher standard than the rest.
The data and technology foundation
The capability can act or reach outside the organisation, and its resistance to manipulation is unestablished
UrgentYou’ve indicated two things that lead to a very high risk when they occur together: the capability acts on its own, connects across systems, or processes content from outside your organisation, and whether it can be manipulated through its inputs has not been established. This is because a manipulated suggestion can be contained when a person reviews it, but a manipulated action the capability takes itself can very often only be undone after the fact rather than contained as part of the process. This is no longer theoretical: in 2025 a state-sponsored group hijacked an AI coding agent to run cyber-attacks largely on its own, and security bodies now report that autonomous agents account for around one in eight reported AI breaches. This very high risk needs to be addressed urgently to create a level of control over an autonomous capability that currently is vulnerable to manipulation.
The legal basis for using personal data here is not pinned down
ImportantYou’ve indicated the lawful basis for using personal data in this capability has not been fully worked through - relied on in general terms, acknowledged as a gap, or not yet established. Under UK and EU data protection law the basis needs to be settled for the specific use, not inherited from the wider business. This looks at the processing rather than the model and where the specific lawful basis is not clear, regulators have shown they will act on it: the Italian regulator fined OpenAI EUR 15 million in early 2025 over data-protection failings including the basis for using personal data. This legal standing should be confirmed and documented before the capability goes further.
Where the data came from is not clearly established
ImportantYou’ve indicated the origin of the data behind this capability is not clearly established - repurposed from another consent, scraped, supplied opaquely by a vendor, or simply unknown. Clear data provenance will need to be shown, documented and proven, if a rights-holder, a regulator, or a customer asks where the data came from. Not being able to trace data is a clear risk - verifying that training and input data was lawfully obtained is increasingly treated as the deploying organisation’s responsibility, not just the vendor’s. Additionally, data that does not have a clear origin is susceptible to errors, either introduced by AI collection or propagated from a data quality issue further up the chain. The data chain should be established and recorded so that it can be assessed and proved if needed.
Nothing decides when this capability’s data and outputs are deleted
ImportantYou’ve indicated that the data and the outputs from this capability accumulate with no defined end point. Storage limitation is a requirement in its own right under UK data protection law, not a housekeeping preference: personal data should not be kept for longer than the purpose needs. The practical exposure is that the volume held grows quietly, so the cost of a breach, a subject access request, or a disclosure order grows with it. Data you no longer hold is data you cannot lose, cannot be asked to produce, and do not have to explain.
Access to this capability’s data and outputs has not been controlled
ImportantYou’ve indicated that access to the data and outputs behind this capability is broadly open or has not been deliberately set. AI tends to create new places where sensitive material comes to rest: prompt histories, output stores, log files, vector databases. These usually inherit whatever permissions they were created with rather than those of the system the data came from, so material that was carefully restricted at source can become widely readable once it has passed through the capability. Where the capability runs on a vendor platform, the same question applies to which vendor staff can see the content.
How the capability was tested is informal or leans on someone else
ImportantYou’ve indicated the capability was tested functionally, informally, on the strength of an upstream supplier, or not yet at all. Functional testing confirms the capability runs; it does not confirm it behaves well on the awkward cases, the edge cases, or the people least like the average. These cases need to be understood and a testing decision made consciously as these are the cases most likely to lead to questions about the credibility and suitability of the AI. What ‘tested enough’ means for this specific use should be defined, the testing against it should be agreed and this decision documented.
Once live, capability monitoring is thin
ImportantYou’ve indicated that monitoring in production is limited to functional health, vendor dashboards, raw logs, or nothing in particular. A capability can run perfectly at a technical level while its outputs drift, degrade, or start landing badly on the people they affect - and functional monitoring will not show you that. Uptime monitoring is easily confused with quality monitoring / observability: the service is ‘up’ while its judgement and outcomes degrade unnoticed. Monitoring needs to include observability: making sure it is clear what the capability is doing and ensuring this is still working as expected and producing reliable, defensible outputs. This requirement should be defined and implemented now.
How data quality is handled is informal or outsourced
Worth notingYou’ve indicated that data quality is handled informally, relied on from upstream, or not yet addressed. A capability is only as reliable as the data feeding it, and quality problems tend to show up as confident, wrong outputs rather than obvious errors - which makes them easy to miss until they have had an effect. This is one of the most common reasons AI projects underperform in practice: the issue is rarely the model, and far more often the data feeding it. Ownership of data quality for this capability should be defined rather than assumed and the relevant standards agreed so that data quality can be shown to be managed and intentional. The reliability of the systems the data comes from belongs in the same assessment: a feed that is accurate but late, intermittent, or unmaintained degrades outputs the same way poor data does.
You cannot fully trace the data the capability used
Worth notingYou’ve indicated that the data flow behind this capability - which data fed which output, and where it came from at each step - is only partly visible or not yet mapped. Lineage is what lets you answer ‘why did it do that’ after the fact. Without it a regulator’s or customer’s question can be hard to answer with confidence because tracing what decisions were made on is mostly guesswork. This gap is often noticed only when an output is challenged when it’s a huge effort to reconstruct the decision path and data relied on. It is far easier to build lineage in now than to reconstruct it later.
The capability rests on a vendor or product that is still maturing
Worth notingYou’ve indicated the capability comes from an emerging supplier, a new or evolving product, or one whose maturity you are not sure of. A maturing capability is not a problem in itself, but it carries more risk of change and less proven track record than one later in its lifecycle: the product can change quickly, support can be thinner, and there is less history to tell you how it behaves over time. This dependency should be managed rather than assumed stable, with a clear view of what you would do if the product or the supplier changed.
The contract may not cover how the capability is actually used
Worth notingYou’ve indicated the contractual position rests on standard terms, terms you cannot fully see, or terms the supplier dictates. Standard software terms often predate the way AI capabilities are used, and can be unclear on the questions that matter here: who is accountable for an output, what the supplier may do with your data, and what happens if the capability causes harm. This can be true for both pure-play AI vendors as we as multi-product providers as it’s easy to rely on old templates. The risk here is that the contract terms are not clear, or absent entirely, about key elements such as liability, data use and AI accountability. These should be clear before they are needed.
The governance picture
The capability acts on its own, without a real check on what it does
UrgentYou’ve indicated the capability does more than suggest or inform - it takes actions - and the human check around those actions is limited or inconsistent. This is very high risk as allowing the AI capability to act without a reliable check is the combination most likely to produce a consequence that is both unexpected and has a negative impact on either the user or the organisation. The more actions the capability can take without being asked, the more this matters, and the recent rise in incidents involving autonomous agents has come largely from capabilities with this risk. An explicit boundary on what the AI capability may do unsupervised should be set.
You might not spot a failure, and the cost of one is not low
UrgentYou’ve indicated two things that create a higher risk together than individually: you would not reliably detect when the capability has gone wrong, and the cost if it does is not low - material or serious on your answers, or not yet established. This risk is that a failure you have no visibility of continues until it is reported to you by an external who has seen an impact, likely a customer or the regulator. This often leads to high impact whatever the actual cost but you have indicated that the cost of failure is not low so this failure could have a big negative impact on cost, reputation or both. A specific detection check for the most costly failure should be put in place.
No one is clearly accountable for this capability
ImportantYou’ve indicated accountability for the capability is collective, unclear, or unassigned. The risk is that shared accountability can become no accountability: when something goes wrong, a capability that was ‘everyone’s’ turns out to be no one’s, and the response stalls while ownership is worked out. The ultimate owner for the capability should be agreed and documented as soon as possible, ideally at both operational and executive level. This will support decision making as well as the right focus for handling problems and improvement opportunities.
Nothing caps what this capability can spend, and no one is answerable for it
ImportantYou’ve indicated both that there is no firm limit on this capability’s running cost and that no single person is accountable for it. Those two gaps compound: a cost with no ceiling needs someone watching it, and a cost with no owner produces no decision when it rises. Spend that climbs is likely to be noticed late, in a finance review rather than an operational one, and by then the question is how to explain it rather than whether to allow it.
Regulatory and reputational lens
How this would look from the outside is unsettled
Worth notingYou’ve indicated some unease about how this project would look if it drew outside attention, or that it has not yet been considered through that lens. Reputational exposure can come from the capability itself but is often from the gap between how it works and how it would be described once something has gone wrong. The risk here is that the potential exposures may be unknown and hence hard to quantify risk levels against.
This capability is likely within scope of the EU AI Act
InformationalYou’ve indicated the capability reaches people or operations in the EU. The EU AI Act applies on the basis of where the people affected are, not where the organisation are based, so a capability run from the UK or elsewhere can still fall within it. This is not a finding about anything being wrong - it is a note that the EU framework is one of the lenses this project sits under, and the obligations scale with how the use is classified rather than applying uniformly. Where this use sits in the Act’s risk tiers is worth establishing rather than assuming. The timing is also split rather than uniform: the transparency obligations, which cover telling people they are interacting with AI and marking AI-generated content, already apply, while the obligations attaching to high-risk uses arrive later in the decade. Establishing which of these the use touches is more useful than treating the Act as a single future deadline.