One architectural insight per week.
Share
Agentic AI: Evaluating Metrics, Data, and Business Impact in Complex Ecosystems
November 5, 2026
4 min read

Marco Zamboni
Chief AI Officer
In a company with ERP, CRM, commerce platforms, and legacy systems, a working demo isn't enough to validate an agentic AI project. The crucial question is: can the agent make correct decisions throughout a multi-step process, using real data and systems and producing a measurable business outcome?
The white paper reports that approximately 95% of generative AI pilots fail to produce a measurable business impact and that, by 2025, 42% of companies will have abandoned most of their AI initiatives. These data refer to the broader context of AI and generative AI, not exclusively to agentic projects, but they highlight the risk of confusing experimentation with the actual value generated.
To evaluate an investment in agentic AI, two elements are therefore required, which must be defined before scaling: a metrics framework capable of measuring end-to-end decisions and results and a verification of the actual readiness of the data on which the agent will operate.
Chief AI Officer
«The value of an AI agent is not measured in the demo but in its ability to achieve a business objective with real data, verifiable decisions and governed autonomy.»
95%
Pilot senza impatto
42%
Companies abandon initiatives
5 sizes
Evaluation of agents
4 requirements
Agent-ready data
The limit of the demo effect: an AI agent is not evaluated like a chatbot
Generative AI produces content from a request. Agentic AI uses generative models as part of a larger system, including planning, memory, use of tools, and goal-oriented execution cycles, under human supervision. Evaluating only the correctness of a response therefore means observing part of the process, not its overall outcome.
The binary metric error. An agent may generate correct text but choose the wrong tool, use incomplete information, or fail to switch between two systems. The evaluation must consider the entire sequence of decisions and actions: goal achievement, resources employed, need for human intervention, and the ability to handle unexpected events.
The data bottleneck. In complex ecosystems, information is distributed across applications, functions, and legacy systems. Access to data alone is not enough: shared definitions, relationships between entities, business rules, and permissions applied during execution are needed.
The risk is remaining in the pilot phase. The paper highlights the gap between the diffusion of AI tools and their economic impact. Vertical use cases, tied to processes such as procurement, supply chain, or customer service, can generate direct value, but require more rigorous integrations and evaluation criteria than a generalist copilot.
From agent performance to data readiness: a framework for decision making
The evaluation begins with a verifiable business objective: reducing request approval time, reducing errors, or limiting manual data entry. This objective is used to build a framework that connects technical performance, operational results, and the conditions required for production operation.
Operational metrics include end-to-end task success, correctness of actions, model cost per activity, latency, hallucination rate, and frequency of human corrective interventions. To verify value before scaling, the document also references business indicators such as cycle time, error rate, cost per interaction, and escalation.
From Evaluation to Rollout:
The proposed path combines the selection of high-value use cases, data verification, governance, involvement of process experts, and infrastructure control. The deployment occurs in phases, with human oversight at the riskiest stages and action tracking to enable auditing, diagnosis, and improvement.
Procurement is the application example developed in the white paper: a purchasing agent must operate on information distributed across requests, sourcing, contracts, payments, and supplier management. It is this continuity of context, along with integration with ERP and external systems, that makes the case study useful for understanding the requirements of an enterprise project.

Written by
Marco Zamboni
Chief AI Officer
He combines strategic vision and technical expertise to translate data, processes, and business objectives into AI solutions. He combines this with curiosity, critical thinking and the ability to lead cross-functional teams to foster responsible AI adoption and build solutions truly aligned with customer needs.
At a glance
What Matters
The essential points before diving deeper: what matters, what doesn't and where real impact is generated.
Download the report- You need to validate the business case for an agentic AI project, not just the performance of a demo.
- You work with ERP, CRM, commerce platforms and legacy systems, with data spread across multiple business functions.
- You want to measure end-to-end outcomes, costs, action accuracy and the need for human intervention.
- You need to verify data access, context, permissions and traceability before deployment.
- You are a CTO, CIO, CDO/AI Lead, Head of Digital or CMO and you need to connect technology decisions to measurable business outcomes.

Bring AI to the content your business needs.
Let's start from your current process to identify a first use case and clarify what is needed to implement it.












