LLM Evaluator (Model Response Analyst)

About the role

We are looking for a detail‑oriented LLM Evaluator to assess and improve the performance of large language models. You will analyze AI‑generated content for accuracy, coherence, factual reliability, bias, safety, and alignment with our guidelines.

Key responsibilities

• Evaluate and rank model‑generated text using complex rubrics covering factuality, coherence, safety, instruction‑following, and creativity.
• Compare multiple model responses to the same prompt and justify the preferred output.
• Provide concise feedback to modeling and training teams about recurring failure patterns.
• Craft adversarial prompts to expose biased, harmful, or insecure outputs and help patch safety gaps.
• Collaborate with QA to refine evaluation guidelines and handle ambiguous edge cases.
• Participate in cross‑checking sessions to calibrate scoring standards and ensure inter‑rater reliability.
• Investigate underperforming model outputs, hypothesise root causes, and flag novel behaviors for the research team.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP/AI quality assurance, or cognitive science.
• Bachelor’s degree in Computer Science or a related field.
• Strong ability to craft prompts that elicit specific model behaviors and to explain why an output is good or bad.
• Experience with Reinforcement Learning from Human Feedback (RLHF) data collection and evaluation consistency monitoring.
• Familiarity with IAA score analysis, calibration sessions, dataset sourcing, cleaning, and annotation for LLM fine‑tuning or evaluation.
• Understanding of data distribution impacts, A/B testing concepts, and experimental design for model comparison.

Required skills

• Prompt engineering
• RLHF data collection
• A/B testing