Скачать или смотреть Testing Custom Generative AI Applications for Safety and Quality

Topic: AI Advanced (TRACK)

Speakers:
Margaret Hagan, Stanford Legal Design Lab
Drew Jackson, People's Law School
Quinten Steenhuis, Suffolk University Law School, Lemma Legal LLC

Generative AI applications hold tremendous promise to address the flexibility, usability, depth, and breadth of legal technology solutions for access to justice problems. However, unlike prior rules-based solutions, generative AI solutions need to be thoroughly tested to ensure consistent and acceptable quality.In this session, we will discuss:The Stanford Legal Design Lab’s framework for evaluating legal AI solutions, and the promise of AI evaluation of AI-answers;Automated testing, iteration and validation of the People’s Law School’s Beagle+ legal help chatbot against a benchmark set of real legal questions;Benchmarking the accuracy of FETCH, a new AI-powered legal issue classifier built by Lemma on behalf of the Oregon State Bar and the Virginia Legal Aid Society against a 200 node taxonomy and more than 400 real legal questions.The session will include a concrete discussion of how to develop evaluation criteria, developing a dataset to benchmark performance, and the tools and strategies that can help validate the safety of an AI solution before releasing it to the general public.