Introducing OSPRay EvalSeptember 21, 2026

Twelve hours to write a ray tracer

We gave four AI models one working day to build a 3D renderer from scratch, then checked their pictures against the real thing. Here is what we measured, how the grading works, and why tests like this matter.

The problem with most AI tests

When a new AI model is announced, it comes with a table of benchmark scores. Many of those benchmarks look like exams: a question, a short answer, a mark. Exams are cheap to run and easy to compare, but they have two weaknesses that grow as models get better.

The first is contamination: the questions leak. Popular benchmarks are discussed, reposted and solved all over the internet, and the internet is a large part of what models train on. If the test questions end up in the training data, a high score partly shows that the model remembers the answers, not that it can work them out. From the outside, it is hard to tell how much of the score is which.

The second is saturation: the benchmark fills up. On several of the most used benchmarks, the leading models now score within a few points of each other, close to the top of the scale. At that point the difference between two models is about the size of the benchmark's own noise, and some of the remaining wrong answers are mistakes in the answer key rather than in the model.

Harder questions help, but only for a while. Each new exam is used up faster than the last, and it still measures one short answer at a time. What exams cannot show is whether a model can carry a real piece of work from start to finish. So alongside them, we assign work: a job that takes a day, that has to hold together as a whole, and that can be checked against reality rather than against an answer key.

What is OSPRay Eval?

OSPRay is a ray-tracing renderer from Intel. You give it a description of a 3D scene, meaning the geometry, the materials, the lights and the camera, and it produces a photorealistic image of that scene. Renderers like this sit behind scientific visualisation and product imagery. OSPRay is large, precise, and took experts years to build.

Our eval asks an AI model to build its own version. The model starts with an empty repository and three things: the API documentation that specifies what each OSPRay function is supposed to do, a few dozen example scenes together with the images Intel's renderer produces for them, and a tool that renders any scene with the real OSPRay, so the model can compare its output as it goes. It gets twelve hours. It may not copy or borrow any existing rendering code. At the end, it hands in a library.

We then load that library into a test harness and run it on a hidden test set of 182 scenes the model has never seen: rooms, glass, metal, fog, hair, animations, and a few scenes with millions of primitives that have to render fast. For each scene, the image from the model's library is compared with the image from OSPRay. The closer the match, the higher the score.

How grading works

Three things are scored. They count for 60, 30 and 10 percent of the final mark.

Images (60%). Each frame the model's library renders is compared with OSPRay's frame, patch by patch. The comparison is tuned to forgive the sampling noise that ray tracing always has, and to catch real mistakes: a wrong colour, a missing shadow, an object that is not there. A perfect match scores 1, a blank image scores 0.

Behaviour (30%). The library also has to behave correctly as an API: return the right error codes for bad input, answer queries such as an object's bounding box, and cancel cleanly when a render is stopped partway through. These are pass-or-fail checks, hundreds of them.

Speed (10%). A few scenes are deliberately enormous. If the model's renderer takes far longer than OSPRay to draw them, it loses points. A renderer that is correct but hopelessly slow does not get full marks.

Two things make the score trustworthy. The test set is hidden, and we confirmed that a library which simply memorised the example scenes would score almost nothing on it. And before any model ran, we ran eleven sanity checks on the grader itself: OSPRay scores a perfect 1.0 against itself, a library that renders nothing scores close to 0, doubling the brightness of every light gets caught, and so on. One of those checks failed the first time and made us fix the grader. That is what the checks are for.

Where the models stand

Each model got one attempt and twelve hours, with the option to stop early if it believed it was finished.

No model reached full marks, which would mean near-perfect images on every scene, almost every behaviour check passed, and every large scene rendered fast enough. The two strongest models are effectively tied, and each will need more than one attempt before we rank them. What the table does show is that the test separates models cleanly, from a floor to a middle to near the top, which is exactly what a public exam can no longer do.

The most useful finding was not the ranking. We expected the models to lose points on the exotic parts of the job and keep the basics. Instead, the strong models built nearly everything and then lost a little of it almost everywhere. That is a different kind of gap, and it is the kind you can only see with a test built like this one.

Why this matters

AI labs improve models by measuring them. If the measurement is an exam that leaks and fills up, some of the improvement it drives is an illusion, and everyone finds out later. If the measurement is a day of real work, checked against a real piece of software, with a hidden test set and a grader that has been checked against itself, then what it rewards is what we actually want: building things that work.

The distance between 0.85 and 1.0 on this test is made of concrete abilities, and it can be measured today. That is what makes it worth having.

← All research