We are looking for an innovative Applied Scientist to join the Books Store team. This role will start with a focus on our centralized auto-evaluation and guardrail platform for AI-powered features across Books. This role is critical in designing and implementing scalable, science-driven evaluation solutions that ensure consistent quality standards across features such as summarization, recommendations, character insights, and more.
As part of this foundational team, you will work closely with engineers, product managers, and other scientists to develop and maintain reusable auto-evaluators, conduct rigorous experimentation, and build systems that help multiple product teams evaluate LLM outputs robustly, quickly, and reliably. You will also contribute to the AI Bar Raiser (AIBR) initiative, which helps uplift evaluation and modeling best practices across the organization.


Key job responsibilities
1. Design, prototype, and productionize auto-evaluation methods (e.g., for faithfulness, coherence, tone, personalization, safety) to assess LLM-generated content at scale.
2. Collaborate with feature teams to adapt and extend evaluation tools to meet evolving feature-specific needs.
3. Develop scientifically rigorous and efficient experimentation frameworks to compare prompt, model, or context configurations.
4. Drive prompt optimization strategies using evaluator feedback, error clustering, and automated suggestion mechanisms.
5. Contribute to defining and implementing the AI Bar Raiser (AIBR) process for design and launch reviews of AI features.
6. Advance the state-of-the-art in LLM evaluation through internal innovation or external research contributions.