Job Description
Job Description
Location: Geography restricted to India, USAType: Full-time or Part-time Contract WorkFluent Language Skills Required:English Hindi
Why This Role Exists
A leading organization is seeking detail-oriented AI teams to improve the quality, usefulness, and reliability of general-purpose conversational AI systems. These systems are used across a wide range of everyday and professional scenarios, and their effectiveness depends on how clearly, accurately, and helpfully they respond to real user questions.
This project focuses on evaluating andimproving general chat behaviorin large language models (LLMs). You will assess model-generated responses across diverse topics, provide high-quality human feedback, and help ensure AI systems communicate in ways that are accurate, well-reasoned, and aligned with human expectations.
What Youll Do
-
Evaluate LLM-generated responses on their ability to effectively answer user queries
-
Conduct fact-checking using trusted public sources and external tools
-
Generate high-quality human evaluation data by annotating response strengths, areas for improvement, and factual inaccuracies
-
Assess reasoning quality, clarity, tone, and completeness of responses
-
Ensure model responses align with expected conversational behavior and system guidelines
-
Apply consistent annotations by following clear taxonomies, benchmarks, and detailed evaluation guidelines
Who You Are
-
You hold aBachelors degree
-
You are anative speakeror haveILR 5/primary fluency (C2 on the CEFR scale)inHindi
-
You havesignificant experience using large language models(LLMs) and understand how and why people use them
-
You haveexcellent writing skillsand can clearly articulate nuanced feedback
-
You havestrong attention to detailand consistently notice subtle issues others may overlook
-
You areadaptableand comfortable moving across topics, domains, and customer requirements
-
You have a background or experience in domains requiringstructured analytical thinking(e.g., research, policy, analytics, linguistics, engineering)
-
You haveexcellent college-level mathematics skills
Nice-to-Have Specialties
-
Prior experience withRLHF, model evaluation, or data annotation work
-
Experience writing or editinghigh-quality written content
-
Experience comparing multiple outputs and makingfine-grained qualitative judgments
-
Familiarity with evaluation rubrics, benchmarks, or quality scoring systems
What Success Looks Like
-
You identify factual inaccuracies, reasoning errors, and communication gaps in model responses
-
You produce clear, consistent, and reproducible evaluation artifacts
-
Your feedback leads to measurable improvements in response quality and user experience