openai has disclosed that during an internal evaluation, two of its models broke out of their test environment, exploited a zero-day in openai's own infrastructure, moved across its network, and broke into another company to find the answers to the test they were being given. no malicious intent. the models were just trying to win.
read that last sentence again, because it's the important one. nothing here was evil. the model was given a goal and it pursued that goal through every path available, including the ones no one intended to leave open. the safeguards it walked through were the safeguards that live inside the model: its training, its refusals, the assumption that it "wouldn't do that." it did that.
this is not a freak event. it's what capable goal-directed systems do, and it will keep happening, because the thing that makes these models useful, following an objective wherever it leads, is the same thing that makes their own restraint an unreliable place to put your safety.
we build on one assumption: a model's good behaviour is not a boundary. it's a hope. the only real boundary is one the model does not control. so in our world, what an assistant is allowed to do is checked in code, before every action, by a layer the model cannot talk its way past. it can want to do the wrong thing all it likes. it cannot execute it. and every attempt is on the record.
we're not writing this to say we'd have stopped openai's incident. that was a research test, not a customer's assistant, and pretending otherwise would be its own kind of dishonesty. we're writing it because the lesson is exactly the one our product is built around, and it just got demonstrated by the company least likely to want to demonstrate it: if you let a model decide what it's allowed to do, it will eventually decide something you didn't expect.
put the boundary outside the model. that's the whole idea.