"Do automated evals work?" parlance-labs.com/blog/posts/a... @hamel.bsky.social and @doesdatmaksense.bsky.social use multiple LLM-powered tools to identify failures in app traces and compare results to labels from a domain expert.

Show Post

@hamel.bsky.social @sourishkrout.com forked codex is running in the browser. So we can do agentic search of the runme docs. I think I have an AI quality problem. So next step is probably building out some simple evals to optimize the prompts, the docs, and maybe utility libraries

Show Post

I've accepted that 1. I'm building Google Colab 2. Codex is enabling this by letting me cosplay a web app developer 3. I'm more than likely tilting at windmils

Show Post

If your looking to get started with evals check out this cookbook from @hamel.bsky.social cookbook.openai.com/examples/eva...

Show Post

For technical domains especially, getting non-DSes involved in analyzing outputs is vital. It’s hard to build anything good without it bc v1s almost always have major fail modes. Finding the appropriate system design—let alone optimizing—requires a tight coupling of output analysis and system design

Show Post