Role Overview

We are seeking a highly experienced Principal AI Systems Engineer to architect, build, and own end-to-end AI automation systems that solve real business problems. The ideal candidate will have deep expertise in Agentic AI, LLM orchestration, prompt engineering, MCP servers, and production-grade AI systems.

This role requires hands-on technical leadership in designing scalable AI platforms, multi-agent systems, evaluation frameworks, and production deployments while ensuring measurable business impact.

Responsibilities

AI System Architecture

Architect and own end-to-end AI automation systems and agent workflows.
Design reusable AI frameworks, internal CLI tools, and agent scaffolding.
Develop and maintain system prompts, agent instructions, and MCP server definitions.
Configure and optimize AI development environments including Claude Code, Codex CLI, and related tools.
Evaluate architectural trade-offs related to scalability, reliability, latency, cost, and maintainability.

AI Pipeline Development

Build production-grade AI pipelines using Python.
Develop orchestration frameworks, structured prompting pipelines, and context assembly mechanisms.
Design and implement multi-agent architectures including orchestrator-worker patterns.
Integrate AI systems with APIs, SDKs, internal platforms, CI/CD pipelines, and enterprise systems.
Implement robust schema validation, error handling, retries, and monitoring capabilities.

Prompt & Context Engineering

Design, version, and maintain system prompts as engineering artifacts.
Develop prompt regression testing and evaluation strategies.
Optimize context windows for performance, cost, and accuracy.
Build and manage knowledge assets including document repositories, rule engines, and validation libraries.
Collaborate with domain experts to translate business knowledge into AI-driven workflows.

Evaluation, Reliability & Monitoring

Build comprehensive AI evaluation frameworks and benchmark suites.
Implement LLM evaluation techniques including LLM-as-a-Judge methodologies.
Monitor production AI systems using tools such as LangFuse, Arize, or equivalent platforms.
Perform failure analysis and continuously improve system performance and reliability.
Define and track key AI productivity and business metrics.

Technical Leadership

Establish engineering standards, architecture patterns, and quality gates for AI development.
Implement governance, audit trails, and version control for prompts and AI artifacts.
Collaborate with product, engineering, and leadership teams on roadmap planning and technical strategy.
Mentor and guide engineering teams on AI best practices.

Requirements

Mandatory Skills

8+ years of software engineering experience with strong expertise in Python.
Proven experience building and deploying production AI systems from scratch.
Strong hands-on experience with Agentic AI and multi-agent architectures.
Experience with Claude Code, Codex CLI, Cursor, or equivalent AI development platforms.
Expertise in designing and deploying MCP (Model Context Protocol) servers.
Strong experience with LangChain, LangGraph, LlamaIndex, AutoGen, or similar orchestration frameworks.
Deep understanding of Prompt Engineering, Context Engineering, and RAG architectures.
Experience designing evaluation frameworks, benchmarks, and quality monitoring systems.
Hands-on experience integrating AI systems with APIs, tools, and enterprise platforms.
Strong knowledge of cloud platforms such as AWS, Azure, or GCP.
Experience with Docker, containerization, and cloud deployments.
Strong understanding of software engineering best practices, testing, logging, and observability.

Preferred Skills

Experience with LangFuse, Arize, OpenTelemetry, or similar monitoring tools.
Knowledge of vector databases and retrieval systems.
Experience with CI/CD pipelines and DevOps practices.
Exposure to MLOps and AI governance frameworks.