Role Overview
Applied AI Engineer to design, build, and test intelligent conversational experiences and AI agents for learner-facing products. You will transform product and instructional requirements into reliable AI behavior across advisory tools, practice simulations, and diagnostic coaching. Working with Product, Engineering, Content Studios, and subject-matter experts, you will ensure AI systems deliver accurate, consistent, and safe interactions. The role covers the full lifecycle—from prompt design and prototyping to evaluation, production deployment, and monitoring—plus technical safeguards for privacy, human oversight, and accreditation. Fully remote, long-term, Agile environment focused on innovative learning technologies for regulated industries.
Responsibilities
- Translate scenario specifications, decision models, and assessment rubrics into prompts, application logic, and testable AI behaviors.
- Identify missing requirements, technical constraints, and risks, escalating to Product, Engineering, and subject-matter experts.
- Define AI model interactions with application services and data sources, including APIs, validation rules, error handling, and permitted actions.
- Design system prompts, structured outputs, and validation mechanisms to prevent invalid responses.
- Build conversational runtimes with controlled personas, session state, and state transitions.
- Implement RAG and grounding from approved knowledge sources, including source references.
- Maintain version control and documentation for code, prompts, schemas, and evaluation assets.
- Run structured experiments with Product teams, defining objectives and success criteria.
- Move validated prototypes into production with Engineering and verify deployed behavior.
- Contribute to technical controls for accreditation and compliance, including issue flagging, review workflows, disclosures, and logging.
- Monitor and evaluate AI performance, identifying improvements in quality, reliability, security, and efficiency.
Requirements
The ideal candidate has hands-on experience developing LLM-powered applications and multi-step conversational workflows, combining expertise in prompt engineering, agentic systems, retrieval, and evaluation with judgment around AI safety, privacy, and human oversight.
- Expert-level prompt and context engineering, including system prompts, few-shot examples, prompt chaining, routing, and task decomposition.
- Strong experience with structured outputs, JSON Schema, function calling, validation layers, and retry/fallback mechanisms.
- Practical knowledge of RAG architecture, including chunking, metadata filtering, hybrid search, reranking, and citation grounding.
- Experience managing token budgets, prompt caching, context compaction, and long-session memory.
- Expertise in agentic systems, including tool use, planner/executor patterns, multi-agent orchestration, state machines, and human-in-the-loop checkpoints.
- Ability to define tool schemas, permission scopes, and action limits for AI agents.
- Experience with AI evaluation methods such as golden datasets, rubric-based scoring, LLM-as-judge, and regression testing.
- Familiarity with LLM observability and tracing.