Role Overview
The Data-AML-Engine Orchestration team builds large-scale machine learning infrastructure that powers online model serving across ByteDance products, including TikTok. You will work on systems affecting GPU utilization, serving latency, infrastructure reliability, and MLE productivity.
Responsibilities
- Design and build foundational orchestration capabilities for machine learning platforms, including Kubernetes Operators, container runtimes, and lifecycle management.
- Build multi-tenant resource and quota systems supporting priorities, preemption, fair sharing, elasticity, and cross-cluster scheduling. Improve GPU utilization and cost efficiency through resource pooling and FinOps.
- Build lifecycle orchestration for online model serving, including model and image distribution, deployment, upgrades, rollback, autoscaling, multi-cluster operation, and disaster recovery.
- Build serving orchestration and traffic management capabilities for disaggregated serving clusters, including topology-aware scheduling, KV Cache affinity, intelligent request routing, and QoS/SLA management.
Requirements
- Currently pursuing a Bachelor's or Master's degree in Computer Science, Software Engineering, Artificial Intelligence, or related technical field.
- Proficiency in at least one of Go, C++, or Python, with solid foundation in data structures, algorithms, and software engineering principles.
- Familiarity with Linux and foundational understanding of operating systems, computer networks, concurrent programming, and distributed systems.
- Strong hands-on and exploratory abilities, willingness to investigate systems through source code, metrics, logs, profiling, and experiments.
- Systematic and quantitative approach to problem solving, defining measurements, testing hypotheses, and validating system improvements.
- Demonstrated ownership and collaboration through coursework, research, internships, open-source contributions, or engineering projects.
Preferred Qualifications
- Experience with Kubernetes, container runtimes, resource scheduling, quota management, multi-tenant systems, or FinOps.
- Contributions to open-source infrastructure projects such as Kubernetes, Volcano, Koordinator, or OpenKruise.
- Experience with model serving systems such as vLLM, SGLang, Triton, KServe, or Ray Serve, or understanding of KV Cache, Continuous Batching, Prefill/Decode disaggregation, or model parallelism.
- Experience with online services, gateways, traffic management, autoscaling, performance optimization, or highly available distributed systems.
- Experience with GPU/NPU programming, heterogeneous resource scheduling, model distribution, or inference performance analysis.