Evaluating Confidence in Generative Output with an Auditor Model | DataRobot Generative AI Use Case

DataRobot solves the GenAI confidence problem by pairing each Generative model with an Auditor model that evaluates the quality of the output. This framework has broad applicability across use cases where accuracy and truthfulness are paramount.

Learn more at http://datarobot.com/platform/generat...

Content:
DataRobot allows you to deliver real-world Generative AI solutions with confidence. Today's highlighted solution is Evaluating Confidence in Generative Outputs. This can be used for really any Q&A chatbot, but especially the ones that need to provide accurate and complete responses. As one executive put it when seeing this example, "This solves the confidence problem."

This diagram shows a user prompt being routed to a Generative AI model that has access to a vector database which has been created in advance. It contains source documents such as technical documentation, academic articles, frequently asked questions, or legal briefs. The Generative AI model creates a response to the prompt, but before showing the response to the user it's routed to a Predictive AI model that serves as the Auditor. The Auditor model evaluates the overall accuracy and alerts the user if there's a high probability that the information is inaccurate. Importantly this Auditor can be trained to detect incomplete or out-of-date responses - two scenarios which are very hard for a human to detect even if they're familiar with the topic. The demonstration that follows shows an end user typing a question to interact with a Generative AI solution that accelerates the process of responding to Requests for Proposal. RFP responses must be accurate and current. In this example, which pairs generative and predictive systems, we have the response to the question, but we also have references and we have scores from the audit model. In this case the audit model believes that the answer is correct and can be trusted. The sub-scores that you see here for the audit model are completely within your control and will definitely vary by use case.

Note that the choice of front end is entirely separate from the solution design. In this case the same solution is presented as both a Slack integration and as a web application. In both cases, the references from the vector database, the audit model correctness score, and all of the sub-scores are presented to the user. This gives the user evidence for and confidence in the generated response. With a platform approach you can confidently build valuable and safe Generative AI applications at enterprise scale. At DataRobot, our extensible platform helps you build quickly and operate securely to bring your Generative AI solutions to life. Learn more at https://www.datarobot.com/platform/ge... where you can also request a personalized demonstration or sign up for a free trial.

Stay connected with DataRobot!
► Blog: https://blog.datarobot.com/
► Community: https://community.datarobot.com/
► Twitter:   / datarobot  
► LinkedIn:   / datarobot  
► Facebook:   / datarobotinc  
► Instagram:   / datarobot