ML Research Scientist, Pegasus

WHO WE ARE At TwelveLabs, we are pioneering the development of cutting-edge multimodal foundation models that have the ability to comprehend videos just like humans do. Our models have redefined the standards in video-language modeling, empowering us with more intuitive and far-reaching capabilities, and fundamentally transforming the way we interact with and analyze various forms of media. With a $110+ million in Seed and Series A funding, our company is backed by top-tier venture capital firms such as NVIDIA’s NVentures, NEA, Radical Ventures, and Index Ventures, and prominent AI visionaries and founders such as Fei-Fei Li, Silvio Savarese, Alexandr Wang and more. Headquartered in San Francisco, with an influential APAC presence in Seoul, our global footprint underscores our commitment to driving worldwide innovation. Our partnership with NVIDIA and AWS gives us access to the most advanced chips, including B300s, enabling us to push the boundaries of what's possible in video AI. We are a global company that values the uniqueness of each person’s journey. It is the differences in our cultural, educational, and life experiences that allow us to constantly challenge the status quo. We are looking for individuals who are motivated by our mission and eager to make an impact as we push the bounds of technology to transform the world. Join us as we revolutionize video understanding and multimodal AI. ABOUT PEGASUS Pegasus is TwelveLabs' core video understanding product, turning video into useful analysis by reasoning over visuals, speech, audio, and on-screen text. The team is not building a generic Video LLM in isolation; we build customer-facing video intelligence workflows that require temporal understanding, structured outputs, and production-grade reliability. A key example is Segment, our time-based metadata capability. Instead of asking the model a broad question about a video, customers define the exact segment types they care about and the metadata fields they want back. Pegasus then finds the relevant start and end times and returns structured metadata for each segment, such as titles, summaries, topics, people, visual subjects, confidence, or domain-specific labels. This is designed for workflows where “what happened” is not enough; customers need to know when it happened and receive metadata that can flow directly into search, archive, editing, compliance, or content management systems. For example, a news archive customer can define a segment type like editorial_narratives and ask Pegasus to split a long broadcast into individual stories. For each story, Pegasus can return a timestamped segment with fields such as segment_title, description, editorial_subjects, visual_subjects, names, and confidence. The output is not just a summary of the full video; it is a structured timeline of the video, aligned to the customer's schema. This is the distinction that matters for Pegasus: general video analysis answers questions about video, while Segment turns video into time-based, structured data tailored to a specific business workflow. Learn more about Pegasus! - Building Structured Video Assets: A Time-Based Metadata (TBM) Pipeline https://www.twelvelabs.io/ko/blog/%EB%B9%84%EB%94%94%EC%98%A4%EB%A5%BC-%EA%B5%AC%EC%A1%B0%ED%99%94%EB%90%9C-%EC%9E%90%EC%82%B0%EC%9C%BC%EB%A1%9C-time-based-metadata(tbm)-%ED%8C%8C%EC%9D%B4%ED%94%84%EB%9D%BC%EC%9D%B8-%EA%B5%AC%EC%B6%95%EA%B8%B0 - Quick Shorts demo: YouTube Shorts https://www.youtube.com/shorts/uaEcQ07AKGg ABOUT THE TEAM The Pegasus team sits at the core of TwelveLabs’ video understanding capabilities and is responsible for driving Pegasus, our Video Analysis product. Our focus is on developing multimodal video analysis systems that are designed for high instruction following capability and producing highly complex, hierarchically structured outputs. We focus on shipping products with real-world value rather than doing research in isolation, and we work in a goal-oriented, cross-functional team that encompasses both ML researchers and engineers. Our work covers a broad range of challenges: large-scale distributed training of multi-modal LLMs that span from pre-training to RL, accurate temporal segmentation and structured metadata extraction for real-world use cases, extending temporal context length to multiple hours, and data curation processes that enable well-aligned evaluation and performance improvements through training data enhancements. Our team has access to the most advanced chips in the world, including NVIDIA B300s, to push the boundaries of video analysis systems—accelerating our research-to-production cycle as fast as possible. This role may be leveled as ML Research Scientist, Senior ML Research Scientist, or Staff ML Research Scientist depending on a candidate’s research depth, technical scope, and track record of impact. IN THIS ROLE, YOU WILL - Define and drive research problems that advance Pegasus’s video