[Professional Contract] SDET AI QA / Evaluation Engineer – Agentic Applications

求人詳細

Our client is establishing a brand-new capability within their AI Platforms Team and are looking for a bilingual AI QA / Evaluation Engineer to build evaluation foundation from scratch.

You will serve as the quality guardian for applications powered by LLMs, RAG pipelines, OCR, tool calling, and automated workflows. This role is perfect for a technical QA professional who is curious about LLM behaviors, proactive about finding edge-case risks, and passionate about continuous automated validation.

Key Responsibilities:
- Build AI Evaluation Datasets: Create and maintain robust "golden datasets" and test cases that act as unit tests for prompts, RAG responses, OCR outputs, and agent workflows.

- Test System Correctness: Validate that agentic systems demonstrate sound reasoning, strict retrieval accuracy, correct tool execution, and reliable fallback handling.

- Mitigate Risk & Hallucinations: Define methodologies to detect production hallucinations, measure answer quality trends, and build operational dashboards (Power BI, Grafana, etc.).

- Automate Evaluation Pipelines: Build reusable test harnesses and integrate automated evaluation/regression checks directly into CI/CD release gates.

- Responsible AI & Compliance: Evaluate applications for potential bias, unfair treatment, compliance issues, and data privacy risks within regulated environments.

- Cross-Functional Partnering: Translate business expectations into measurable technical criteria alongside engineers, data scientists, and business stakeholders.

こんな方を求めています

経験・スキル

Profile & Technical Requirements:
- Min of 4- 5 years of work experience
- Core experience in Software QA, test automation, data validation, or AI/ML application quality engineering.
- Solid conceptual understanding of LLMs, prompt engineering, RAG, OCR, tool calling, and agentic system behaviors.
- Hands-on automation scripting experience using Python or JavaScript/TypeScript.
- Working familiarity with modern CI/CD pipelines, Azure DevOps, and GitHub Enterprise.
- Familiarity with testing frameworks, observability tools, or dashboarding software (e.g., pytest, Playwright, LangSmith, PromptFlow, MLflow, Power BI, Grafana) is a major plus.
- Business-level Japanese (or above) is mandatory for this role, as you must read Japanese business context documents and align directly with local teams.

学歴

大学卒業以上

募集要項

職種

品質管理・テスト・評価

給与

年収 700 〜 1,200万円

賞与

-

雇用形態

契約社員

雇用期間

期間の定めなし

試用期間

無し

就業時間

09:00～18:00

休日休暇

土曜日　日曜日　祝日

保険

健康保険　厚生年金保険　雇用保険

受動喫煙防止措置

敷地内禁煙

特長

外資系企業

業種

保険

※詳細はお仕事のご紹介時にお伝えします