Fiddler AI urges operational visibility and guardrails for generative and agentic models
Loading...
Summary
Fiddler AI demonstrated an AI observability platform, warned about hallucinations and bias in deployed models, and urged human oversight, auditing and governance for federal AI uses such as fraud detection and national‑security workloads.
Amit Parker, co‑founder and COO of Fiddler AI, told the Oversight roundtable that agencies deploying predictive, generative and agentic AI need a "purpose built neutral AI command center" to provide operational visibility, detect hallucinations and prevent bias.
Parker said deployed models can be opaque and produce incorrect assertions called "hallucinations," and he showed a sonar image where a model mislabelled a plane on the seabed as a ship because it focused on a sonar shadow. "If AI is not trained and deployed correctly, you can also introduce bias in decisions which can have an outsized impact," Parker said, citing credit lending as an example and recommending fairness metrics and audit trails to comply with statutes such as the Equal Credit Opportunity Act and the Fair Housing Act.
Fiddler demonstrated dashboards that track model accuracy, data drift, disparate‑impact metrics and per‑interaction audit trails. The platform flags potential jailbreaks and hallucinations in real time and provides explainability tools (for example, colorized pixel maps showing which image areas influenced a model’s output). Parker said Fiddler works with enterprise and federal clients, including a named Navy program working on seabed imagery, and aligned its recommendations to five ethical AI principles.
Committee members asked whether human oversight could persist as models become more autonomous; Parker said human trust is necessary for sustained adoption and that observability is a prerequisite for safe scaling. Other speakers, including Anthropic representatives, agreed on the need for third‑party testing and governance structures.
Less critical details: Fiddler described specific dashboard features (trace/span views for agentic workflows, root‑cause analysis and real‑time alerting) and said the company has worked with large financial institutions to surface and remediate bias in production models.
