Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)

In this podcast episode, Hamil Hussain and Shreya Shankar share their insights on the increasing importance of evaluations (evals) in AI product development. They begin by emphasizing that to build great AI products, developers need to excel at creating evals, which they deem a high ROI activity. Contrary to some opinions, evals aren't solely based on AI's self-assessment; rather, they require human oversight.

The hosts dive into the foundational definitions of evals, illustrating their relevance through practical examples, such as a real estate assistant application. They outline a systematic process called error analysis, where product managers and engineers evaluate AI outputs to identify problems and improve the application's performance. The episode clarifies that evals should not be thought of merely in terms of formal tests or checkboxes; they encompass various methods of ensuring quality, including the use of LLMs (Large Language Models) as judges.

Through discussions on how to create metrics that track application performance, they outline best practices for conducting evals effectively. Hussain and Shankar stress the importance of looking at raw data and feedback loops in developing products that truly resonate with users, inviting listeners to embrace the learning journey that comes with doing evals. They stress that this process is transformative and offers significant insights that can lead to actionable improvements in AI systems.