Senior Software Engineer

Responsibilities

• Design and ship end-to-end features spanning data models, APIs, background workers, and UI.
• Build scalable systems for running, scoring, aggregating, and comparing agent and model evaluation trials.
• Improve benchmark task quality using verifiers, rubric-based grading, and LLM-as-a-judge evaluation.
• Integrate new models, agents, and benchmarks through generalizable systems rather than one-off implementations.
• Maintain performance and reliability through efficient queries, safe migrations, and load-resistant concurrency.
• Review pull requests and help maintain consistent engineering standards, tests, and error handling.

Requirements

• 5+ years of experience building and operating production backend systems, including ownership of at least one service from design through on-call.
• Bachelor’s degree or higher in Software Engineering, Computer Science, or a related field.
• Strong Python, Django or a similar framework, PostgreSQL, and task-queue experience such as Celery.
• Hands-on experience with LLM evaluation, including benchmark construction, agent-trial scoring, and result aggregation and comparison.
• Ability to identify ambiguous instructions, weak or gameable verifiers, leaked answers, and invalid checks.
• Comfort working with TypeScript and React when frontend changes are needed.
• Experience with sandboxed or containerized agent environments such as Docker, Kubernetes, Modal, or Daytona is preferred.
• Experience with LLM-as-a-judge or rubric-based grading, coding agents, MCP, distributed scheduling, concurrency limits, database locking, or open-source evaluation and infrastructure projects is preferred.

Benefits

• Work at the frontier of AI with leading AI labs and contribute to datasets, reinforcement-learning environments, and benchmarks.
• Opportunities to contribute to AI research and showcase work at conferences such as ICLR, ICML, and NeurIPS.
• Collaborate with colleagues with experience at Google, Meta, Amazon, and other leading technology companies.
• Startup-paced environment with significant ownership and exposure to enterprise AI applications.
• The posting states base salary plus equity.