Benchmark · Code to video
VideoSWE
Code that renders a narrated chapter from a course's sources, scored by a rubric judge.
VideoSWE is not released yet.
01The tasks
Input is a course's sources; output is one narrated chapter as a Remotion project. The verifier renders it and gates on the basics (builds, one to two minutes, narrated, cites the sources). A rubric judge then scores the teaching.
One narrated chapter per task, built as a Remotion project from the tutor's sources, rendered and judged.
02Reward and method
- Environment
- Programmatic video from source documents, judged by expert educators. No network.
- Verifier
- Gate on the basics, then a rubric judge, reward 0 to 1.
- Score
- Mean reward, averaged over tasks.
- Interval
- 95% bootstrap over tasks. Shared rank unless intervals separate.