AI Infrastructure Operations Engineer

About The Role

We are seeking a highly skilled and experienced AI Infrastructure Operations Engineer to manage and operate our cutting-edge machine learning compute clusters. These clusters would provide the candidate an opportunity to work with the world's largest computer chip, the Wafer-Scale Engine (WSE), and the systems that harness its unparalleled power. 

You will play a critical role in ensuring the health, performance, and availability of our infrastructure, maximizing compute capacity, and supporting our growing AI initiatives. This role requires a deep understanding of Linux-based systems, containerization technologies, and experience with monitoring and troubleshooting complex distributed systems. The ideal candidate is a proactive problem-solver with expertise in large-scale compute infrastructure, dependable and an advocate for customer success.  

Responsibilities

• Manage and operate multiple advanced AI compute infrastructure clusters. 
• Monitor and oversee cluster health, proactively identifying and resolving potential issues. 
• Maximize compute capacity through optimization and efficient resource allocation. 
• Deploy, configure, and debug container-based services using Docker. 
• Provide 24/7 monitoring and support, leveraging automated tools and performing hands-on troubleshooting as needed. 
• Handle engineering escalations and collaborate with other teams to resolve complex technical challenges. 
• Contribute to the development and improvement of our monitoring and support processes. 
• Stay up-to-date with the latest advancements in AI compute infrastructure and related technologies. 

Skills And Requirements

• 6-8 years of relevant experience in managing and operating complex compute infrastructure, preferably in the context of machine learning or high-performance computing. 
• Strong proficiency in Python scripting for automation and system administration. 
• Deep understanding of Linux-based compute systems and command-line tools. 
• Extensive knowledge of Docker containers and container orchestration platforms like k8s and SLURM. 
• Proven ability to troubleshoot and resolve complex technical issues in a timely and efficient manner. 
• Experience with monitoring and alerting systems. 
• Should have a proven track record to own and drive challenges to completion. 
• Excellent communication and collaboration skills. 
• Ability to work effectively in a fast-paced environment. 
• Willingness to participate in a 24/7 on-call rotation. 

Preferred Skills And Requirements

• Operating large scale GPU clusters.
• Knowledge of technologies like Ethernet, RoCE, TCP/IP, etc. is desired.
• Knowledge of cloud computing platforms (e.g., AWS, GCP, Azure).
• Familiarity with machine learning frameworks and tools.
• Experience with cross-functional team projects. 

Location 

• SF Bay Area.
• Toronto, Canada.
• Bangalore, India.