Position Summary...

What you'll do...

Key Responsibilities

AI Evaluation & Transparency

Manage dashboards connecting capability metrics with business metrics to surface what’s working and what isn’t.

Maintain golden datasets for benchmarking model relevance, execution accuracy, personalization, and compliance.

Partner with AI Tech and Tech Ops to calibrate LLM-as-Judge and ensure automated evaluation remains aligned with human judgment.

Shopper Experience Quality & Audits

Conduct periodic audits of shopper-facing AI experiences (answers, nudges, recommendations, in-shop chat).

Define and monitor KPIs for intent recognition, execution accuracy, and personalization precision.

Run cross-journey evaluations to identify friction or divergence patterns.

 

SOP Governance & Continuous Improvement

Own and evolve SOPs for annotation, labeling, and evaluation to reflect changing features and customer contexts.

Capture and codify new edge cases into updated labeling guides.

Ensure all SOPs align with shopper-first outcomes and compliance .

 

Human + AI Divergence Monitoring

Track variance between human labels, automated judgments, and live model outputs.

Create structured escalation paths for high-risk failures (irrelevant, unsafe, or biased responses).

Quantify divergence trends and drive remediation with Product and Engineering.

 

Vendor & Partner Management

Manage external labeling partners, ensuring throughput, accuracy, and compliance SLAs.

Deliver training grounded in real Sparky interactions and golden dataset examples.

Audit vendor output and hold partners accountable for quality metrics.

Closed-Loop Feedback & Model Integration

Build and manage the continuous feedback loop that connects:

Shopper signals

Human review & annotation

Automated LLM evaluation

 

Ensure that every signal from real interactions feeds directly into model training, experience design, and KPI improvement—driving measurable gains in personalization, nudge effectiveness, and conversational trust

Qualifications

2–4 years in AI operations, ML evaluation, or eCommerce data quality; experience managing annotation or human-in-the-loop programs preferred.

Working knowledge of AI evaluation frameworks (LLM-as-Judge, golden datasets, A/B testing).

Skilled in KPI design, audit processes, and vendor oversight.

Analytical storyteller able to link model metrics to business outcomes.

Obsessed with customer trust, transparency, and continuous improvement.

 

Act with Integrity: Follows the law our code of conduct and company policies and sets expectations for others to do the same promotes an environment where associates feel comfortable sharing concerns and reinforces our culture of non retaliation listens to concerns raised by associates takes action and encourages others to do the same holds self and others accountable for achieving results in a way that is consistent with our values

Act with Integrity: Acts as an altruistic servant leader and is consistently humble self-aware honest and transparent

Serve our Customers and Members: Delivers expected business results while putting the customer member first and consistently applying an omni merchant mindset and acts with an everyday Low-cost mindset to drive value and every day Low Prices for customers members

Serve our Customers and Members: Adopts a holistic perspective that considers data analytics customer member insights and different parts of the business when making plans and shaping the teams strategy

Strive for Excellence: Consistently raises the bar and seeks to improve demonstrates curiosity and a growth mindset seeks feedback asks thoughtful questions fosters an environment that supports learning innovation and learning from mistakes and intelligent risk-taking and exhibits resilience in the face of setbacks

Strive for Excellence: Seeks and implements continuous improvements and encourages the team to leverage new digital tools and ways of working

At Walmart, we offer competitive pay as well as performance-based bonus awards and other great benefits for a happier mind, body, and wallet. Health benefits include medical, vision and dental coverage. Financial benefits include 401(k), stock purchase and company-paid life insurance. Paid time off benefits include PTO (including sick leave), parental leave, family care leave, bereavement, jury duty, and voting. Other benefits include short-term and long-term disability, company discounts, Military Leave Pay, adoption and surrogacy expense reimbursement, and more.

You will also receive PTO and/or PPTO that can be used for vacation, sick leave, holidays, or other purposes. The amount you receive depends on your job classification and length of employment. It will meet or exceed the requirements of paid sick leave laws, where applicable.

For information about PTO, see https://one.walmart.com/notices.

ㅤ

ㅤ

ㅤ

ㅤ

‎ 

Minimum Qualifications...

Outlined below are the required minimum qualifications for this position. If none are listed, there are no minimum qualifications.

Bachelor's degree in information technology, computer science, or related area and 5 years’ experience in eCommerce merchandising, site operations, business management, or related area OR 7 years’ experience in eCommerce merchandising, site operations, business management, information technology, computer science or related area.
2 years’ supervisory experience.

Preferred Qualifications...

Outlined below are the optional preferred qualifications for this position. If none are listed, there are no preferred qualifications.

ECommerce Merchandising, Site Operations, Business Management, or related area, Master's degree in Information Technology, Computer Science, or related area

Primary Location...

702 Sw 8Th St, Bentonville, AR 72716, United States of America

Walmart and its subsidiaries are committed to maintaining a drug-free workplace and has a no tolerance policy regarding the use of illegal drugs and alcohol on the job. This policy applies to all employees and aims to create a safe and productive work environment.