Russell Tobin & Associates is currently seeking a Hybrid Data Model Quality Analyst to work for our client in the AI/tech industry. Apply now for consideration.
Contract: W2, 6- Months
Location: Hybrid Tues-Thurs On-site in San Jose, CA
Pay Range: $75-85/Hour depending on experience
Responsibilities:
- Ensure data quality during task development for human-created content used in training Large Language Models (LLMs).
- Define detailed data requirements based on model goals and analysis of model loss patterns.
- Design high-quality, nuanced instructions for expert content creators with illustrative examples.
- Rapidly iterate with content-production vendors to improve data quality in small batch reviews.
- Develop benchmark datasets to assess and validate data quality.
- Benchmark vendor outputs against competitor models using both quantitative and qualitative methods.
- Analyze data from vendor teams and optimize rater pools for quality.
- Audit datasets, identify quality issues, and help develop tools to streamline data review processes.
- Use experimentation to demonstrate data impact and uncover model blind spots.
- Generate regular stakeholder reports on data quality metrics, iteration times, and vendor performance.
- Author and maintain best practices documentation and instructional guidelines.
Requirements:
- Bachelor's degree in English, Creative Writing, Linguistics, Science, Computer Science, or a related field.
- 3+ years of experience in Technical Writing, Quality Assurance, or a related field.
- Excellent writing and editing skills with adaptability to varied styles and tones.
- Strong analytical and problem-solving skills.
- Ability to work independently and collaboratively across technical and non-technical teams.
Preferred Qualifications:
- PhD in a relevant field (English, Linguistics, Computer Science, etc.)
- 5+ years experience in relevant roles with exposure to LLM development or content creation at scale.
- Experience with data benchmarking, quality auditing, or content vendor management.
- Familiarity with model evaluation techniques and qualitative dataset analysis.
Russell Tobin/Pride Global offers eligible employees comprehensive healthcare coverage (medical, dental, and vision plans), supplemental coverage (accident insurance, critical illness insurance, and hospital indemnity), a 401(k)-retirement savings plan, life & disability insurance, an employee assistance program, identity theft protection, legal support, auto and home insurance, pet insurance, and employee discounts with preferred vendors.
LI-JJ2