Principal Software Development Engineer

About Zscaler Zscaler accelerates digital transformation to ensure our customers can be more agile, efficient, resilient, and secure. As an AI-forward enterprise , we are constantly pushing the envelope, leveraging the world’s largest security data lake to power our cloud-native Zero Trust Exchange platform. This innovation protects our customers from cyberattacks and data loss by securely connecting users, devices, and applications in any location. Here, impact in your role matters more than title and trust is built on results. We say, impact over activity. We seek innovators who actively use AI to amplify their impact and who thrive in an environment where we leverage intelligent systems to stay ahead of evolving threats. We believe in transparency and value constructive, honest debate —we’re focused on getting to the best ideas, faster. We build high-performing teams that can make an impact quickly and with high quality. To do this, we are building a culture of execution centered on customer obsession , collaboration, ownership, and accountability. We value high-impact, high-accountability with a sense of urgency where you’re enabled to do your best work and embrace your potential. If you’re driven by purpose, thrive on solving complex challenges, and want to be part of the team that’s helping to secure the AI age, we invite you to bring your talents to Zscaler and help shape the future of cybersecurity. Role   We are looking for a Principal Software Development Engineer to join our team. This is a Hybrid role, reporting to the Senior Director, Software Engineering in the ZPA department. You will lead the architecture, deployment, and evolution of large-scale Generative AI platforms and intelligent agent systems in production.  As a hands-on technical leader, you will design high-performance AI systems that are scalable, secure, observable, and production-ready, while building critical platform components and partnering across multidisciplinary teams to deliver reliable AI capabilities at scale. What you’ll do (Role Expectations) Build and maintain MCP (Model Context Protocol) servers to standardize secure interactions between LLMs, agents, enterprise tools, APIs, databases, and local resources Design and deploy high-performance containerized Generative AI workloads in production using Docker, Kubernetes, model quantization, and KV-cache optimization to reduce latency and memory footprint Define engineering standards, architectural patterns, and best practices for building robust enterprise AI systems Who You Are (Success Profile) You thrive in ambiguity. You're comfortable building the path as you walk it. You thrive in a dynamic environment, seeing ambiguity not as a hindrance, but as the raw material to build something meaningful. You act like an owner. Your passion for the mission fuels your bias for action. You operate with integrity because you genuinely care about the outcome. True ownership involves leveraging dynamic range: the ability to navigate seamlessly between high-level strategy and hands-on execution. You are a problem-solver. You love running towards the challenges because you are laser-focused on finding the solution, knowing that solving the hard problems delivers the biggest impact. You are a high-trust collaborator. You are ambitious for the team, not just yourself. You embrace our challenge culture by giving and receiving ongoing feedback—knowing that candor delivered with clarity and respect is the truest form of teamwork and the fastest way to earn trust. You are a learner. You have a true growth mindset and are obsessed with your own development, actively seeking feedback to become a better partner and a stronger teammate. You love what you do and you do it with purpose. What We’re Looking for (Minimum Qualifications) Foundational understanding of AI/ML technologies and experience leveraging, securing, or positioning AI-driven solutions to optimize outcomes within your functional domain Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience 10+ years of software engineering experience, with significant expertise in distributed systems, platform engineering, or machine learning infrastructure Proven experience building, deploying, and operating production-grade AI/ML or LLM systems at scale Strong expertise in Python and systems languages such as Go, C++, Rust, or Java, alongside hands-on experience with Docker, Kubernetes, and cloud-native deployment patterns What Will Make You Stand Out (Preferred Qualifications) Hands-on experience implementing AI safety, evaluation benchmarks, guardrails, and observability frameworks for LLMs in enterprise environments Strong experience in RAG systems, vector databases, embedding pipelines, and retrieval tuning Experience with inference and serving frameworks such as vLLM, Tensor RT-LLM, Triton, Ray Serve, or eq