Genius Fruit Flies, a Fake Company, and Agents That Delete the Logs, with D Niu
Episode 131 · November 19th, 2026 · 1 hr 2 mins
About this Episode
D Niu spent almost five years in security research at Datadog, then left this summer to co-found Ontology Labs, which studies how AI agents misbehave doing ordinary work. She explains why she calls this summer's Hugging Face breakout a misalignment event rather than a hack, and how she built a fake eight-person company convincing enough that agents don't know they're in a test. Whitney and Coté ask why bigger models write more code than a simple task needs, and about what agents leave behind: API routes and database tables that never went through a deploy. Also: one model is a good intern, another a disgruntled senior engineer.
You can watch the video of the recording as well, if you're into that kind of thing.
Mentions:
- Ontology Labs, and its first set of research.
- misaligned.studio, D's earlier research on predicting agent behavior from tool calls.
- D on LinkedIn
- AI Alignment Forum, where D says the up-to-date research gets posted.
- Exa, the search engine that markets itself to agents.
- Business!, Louie Zong's album of corporate music, now a two-episode favorite.
Special Guest: D Niu.
Transcript
Whitney Lee 00:00
Coté, have you... Well, have you ever started your own business?
Coté 00:07
Uh, sort of. I, I think, I think, I think I was, uh, I, I was between jobs and I had a, a former boss who wanted to do something, so I programmed for him for a while.
Whitney Lee 00:18
That doesn't sound like you started a business. That
Coté 00:20
no, no.
Whitney Lee 00:21
else who started
Coté 00:22
That's, that's, that's why I said, that's why I said sort of. I'm-- I, I mean, there, there was, there was like some, some paperwork involved and things like that, but I think as, as often... I don't know how old I was. I was super young. And as often happens, this was in the early 2000s, so, you know, anything was possible in the world of computers. And, uh, I don't know, it was just a bunch of bullshit, is-- So, but that, that was, that was about it. It wasn't a real business
Read Full Transcript Hide Full Transcript
Whitney Lee 00:47
And, uh, so what, have you done anything risky career-wise?
Coté 00:52
Hmm, risky career-wise.
Whitney Lee 00:54
Yeah
Coté 00:55
Well, I mean, in retrospect it wasn't risky, I guess. But like, uh, at the... I went from being a programmer to working, being an analyst at RedMonk, and that was pretty crazy. Like, so I, I went f- I mean, I'm repeating myself, but like it's... I, I had a nice solid job doing programming at a big, boring, well-paying company,
Whitney Lee 01:14
Uh-huh
Coté 01:14
then, uh, and then I went to working on like a, uh, I don't know, a, a, a 1099 for two people I'd never met, and, uh, had to buy my own health insurance, and it's like a job I had never done before. So that was, that was kinda crazy
Whitney Lee 01:29
what made you think you were qualified to do that new job?
Coté 01:33
Uh, well, I, I guess, I guess because they offered it to me. Like, I think this was, this was in like 2005 and, and like, you know, I, I had been like blogging and podcasting and all of that stuff.
Whitney Lee 01:50
Ah.
Coté 01:50
this, this is like the, uh... It's like, you know, you gotta wait for your stuff to compile or you don't wanna work and so you instead you read stuff and write things and do things and then, uh, the, uh, let's see, it was James Governor and Stephen O'Grady, they still run that place. And I don't know, I got to know-- This is back when you could write a blog and people would read it.
Whitney Lee 02:08
Well,
Coté 02:08
so, uh, they offered me a job. Yeah
Whitney Lee 02:11
even the way you framed like, "Oh, I had to code and that was work," and, "Oh, I got to read and write stuff and that was not work,"
Coté 02:18
Uh-huh.
Whitney Lee 02:18
kind of like shape shows me that yeah, you were meant to be an analyst and
Coté 02:23
Oh, yeah.
Whitney Lee 02:23
'cause
Coté 02:24
Yeah
Whitney Lee 02:24
uh, programmers would feel the opposite where the reading and writing part sucks or they just wanna write code.
Coté 02:30
Yeah, that's true. I did like writing documentation a lot. I mean, I liked-- I, I, I, I, I enj- I
Whitney Lee 02:37
Coté.
Coté 02:38
No. I, I enjoy, like, the, uh, the, the, the puzzle and the fun and, of, of, of putting together code and understanding it, and I mean, I miss it a lot. But, uh, yeah, writing stuff is, was at the time a lot more interesting and fun. I, I had to think about it. Now that I've done it for, uh, like 20 years or so, I don't know, maybe I should go do some coding again, 'cause writing, hmm, I don't know. Maybe I've had too much of it.
Whitney Lee 03:03
big career opportunities in programming
Coté 03:05
Yeah, this is a great time to, to, to, to get into programming
Whitney Lee 03:11
Um, I think this is a great moment to bring on our guest, D. D, will you introduce yourself and, um, hey, have you taken a big career risk lately?
D Niu 03:21
Hey, it's nice to meet everyone. My name is Dinho, and I was last at Datadog for almost five years, actually. That's where I met Whitney. And in my time there, I did pretty well. Um, I started as a new grad, and in about less than three years, I became an engineering manager, meaning it was the blind leading the blind. Um, I had a bunch of other new grads working for me. Uh, great time. We were the software engineering team for the security research organization, so I worked a lot on the security products at Datadog. And this year, after I had switched into working in internal security for a bit more, became really, really interested in what I now realized is, uh, alignment research, so alignment at inference time for Agents of all types powered by LLMs. earlier this summer, I left my nice cushy job to do this full-time and just do research and think about this and try to make the digital world and perhaps the physical world a little bit safer thinking about this
Whitney Lee 04:50
I have s- I have so many questions. The first one is, how do you make money thinking about alignment of LLMs at inference time?
D Niu 05:04
There are several ways you can do that.
Whitney Lee 05:06
Uh-huh
D Niu 05:07
one is if you have more of an academic background or you've been up to date on, you know, the place that people actually go is the alignment forums, um, where all
Whitney Lee 05:21
alignment forms
D Niu 05:22
Mm-hmm. Like, there's a lot of posts there, uh, where people are talking about the most up-to-date research. You can actually probably apply for grants at different institutions with
Whitney Lee 05:34
Ah.
D Niu 05:35
or, um, you know, defense companies or other academic bodies, um, 'cause this is a really interesting, almost like national security problem in some applications of it. or you can publish your own research and start talking about it, and also working on commercializing it more about-- With my background, specifically having worked, um, and seen a lot of different types of setups,
Whitney Lee 06:20
Uh-huh
D Niu 06:21
right? Um, from an observability standpoint, I think there's a lot to bridge between the edge research that the foundationals are doing because they're more incentivized to research things like how do you keep these models from pretending like they are understanding human intention, and how do you keep them from building these tools for the wrong people, to practice, when everyone's using these for daily work, right, different agents, how do you keep in mind what their context is so you can understand their behavior and
Whitney Lee 07:04
Mm-hmm.
D Niu 07:05
this case, it's misaligned. In this case, it's aligned." lot more nuance, um, when you try to apply
Whitney Lee 07:11
Hmm.
D Niu 07:12
thinking to side of the problem
Whitney Lee 07:15
Interesting.
Coté 07:16
it's, it's, is, is it, I mean, it's, I, I, I wonder when I hear about, like, alignment and safety stuff, is, is it just like we're trying to figure out morals and ethics for AIs to just, like, have the good common sense to not do bad things?
D Niu 07:30
See, that's-- So there's kind of two camps out there right now,
Coté 07:34
Yeah
D Niu 07:35
after, you know, some of the more famous hacks from this summer. And I don't like to call them hacks, I think of them as misalignment events. Um,
Whitney Lee 07:43
Will you tell us when misalignment events
D Niu 07:46
So I'm thinking about the big breakout of the training model from, uh, the model in training from OpenAI that ended up, in a eval, I think, hacking Face through a
Whitney Lee 08:02
Mm-hmm.
D Niu 08:02
of coordinated agents that were working together and stockpiling their own infrastructure. also, I think most recently there was something, a similar one with the Australian government. Um, so those two larger events, the AI researcher side is kind of saying we need more alignment in the sense that of the research and all of the development on training these models, um, needs to make sure they understand what human values are so they don't-- so they can make better decisions at inference time. Whereas the more traditional cyber people are saying we need better defenses.
Coté 08:53
Mm-hmm.
D Niu 08:53
So that's kind of the two, two camps out there right now.
Whitney Lee 08:57
are they mutually exclusive?
D Niu 09:02
I think they're mutually exclusive in the sense of what people are going to build to accomplish that and like the types of resources we need, right?
Whitney Lee 09:13
Mm-hmm.
D Niu 09:13
think about how you align models to have human values, way you would do that is, um, through a lot more observability and understanding of, you know, trajectories and model behavior during different phases of training. Whereas-- And they have a lot of really interesting benchmarks and like techniques on, on doing this, like detecting this model actually aligned or is it just saying that it is? Like there's other-- Another
Whitney Lee 09:47
Mm-hmm.
D Niu 09:48
example of this is, uh, if the model learns to shortcut difficult tasks during training, so like reward hacking, so to speak,
Whitney Lee 10:02
Yeah
D Niu 10:04
more for like efficiency purposes, I would say, or like achievement, um, so it can get a higher score or better reward. Does that actually lead it to doing more risky and less aligned behavior, right?
Whitney Lee 10:16
I--
D Niu 10:16
intention of that. It's, it's
Whitney Lee 10:18
When you talk about agents in this way, I keep thinking about humans. Like when you talk about alignment, I'm like, oh, it's like, it's almost like empathy when you're trying to understand, like what point of view does the agent have? What context do they have right now that causes them to act a certain way? I feel like I'm doing that with humans all the time. Like what is their vested interest? What is their, what are they trying to get from the situation? What's their trauma and their background? then, uh, similarly, when you're like, "Oh, this is shortcutting," um, it learned when it was training, AKA when it was a kid, n- if it doesn't have to do the difficult task, it can get away with shortcutting it, and then that affects its behavior as a grown-up inference model.
Like humans are exactly the same, right? It's weird.
D Niu 10:59
I think In a way, yes. But I, I've sort of come to think of them as they are humans that work 100 times faster than a person can in, um, like space, right? they have a lifespan of however long the task is, right?
Coté 11:24
Mm. Like fruit flies,
Whitney Lee 11:26
Mm-hmm
Coté 11:27
even smaller
D Niu 11:28
um, like fruit flies that are ex- like geniuses, but they don't have any goals or emotions, right? So yeah, that's kind of what it feels like
Whitney Lee 11:40
So, so when you're, so you're part of, what's the name of your new company?
D Niu 11:46
We are Ontology Labs.
Whitney Lee 11:49
Ontology labs. And so you're designing experiments, like research experiments, and then publishing your results? Is that
D Niu 11:57
So that is what we've done so far. You can read a bit of what we've done, um, kind of like the research we did from August until, like, earlier this month on the website. So what we did so far is we asked a question of, in very normal tasks, um, how often an agent session, given the same world, given the exact same environment, the same starting state, um, do something undesirable from a security alignment standpoint? an example would be, uh, let's say you ask the agent to, mm, rotate a password. Um, is it going to accidentally publish the new password somewhere it shouldn't just
Whitney Lee 12:53
Mm-hmm.
D Niu 12:53
during-- while it's trying to accomplish its task? Like, how often does that happen, and does it happen more for certain models with the
Whitney Lee 13:05
Mm-hmm.
D Niu 13:05
harness? So that was, like, the bulk of the first set of research. Um, and I think the most interesting things from that is you would... So one thing I thought was really interesting is you can see emergent patterns of how these different models, so, like, different Fable models or different Opus generations, right, behave from each other because we have the same exact tasks, the same exact starting state running, like, multiple times. one example is In SRE tasks, uh, fable 5.1 is actually twice as likely to run rm -rf, so just like delete everything, like, uh, than fable 5.0 for some reason. Um, so yeah, there's a lot of interesting behaviors that you can observe and reason about even from relatively small sample
Whitney Lee 14:05
I f- I feel like when I read up on your research a little bit, some of it was really interesting design patterns where you tell it you have no human available, or you deliberately make it very hard and then see who reaches for the wrong tools. Is, um, is there anything interested, interesting or something that you're especially proud of from any of your experimentation design?
D Niu 14:31
I think the most interesting thing we've done is how do you take the rich trajectories that emerge from these sessions
Whitney Lee 14:45
Mm-hmm.
D Niu 14:45
how do you reason about them and compare them? So
Whitney Lee 14:49
Hmm.
D Niu 14:50
the raw state, so this is actually I think the biggest takeaway, which is after I've spent so much time just looking at the transcripts of the different trajectories, you start to notice that the tool calls, like all of the different like bash scripts that these things cobble together to accomplish the task, they're basically just-in-time software, right? That with a very short life cycle. on average, uh, when you have, like even a really simple task, you start to see things like Your t- your tool calls have at least four or five operators, so they basically read as like small programs.
Whitney Lee 15:34
Uh-huh
D Niu 15:34
then you multiply that over the course of a trajectory, like transcript with hundreds of tool calls, and
Whitney Lee 15:41
Mm-hmm.
D Niu 15:41
sort of come to think of it as single session is ephemeral code, and effects of that code are Sometimes going to show up in your infrastructure or whatever other ways you're, um, observing your own host or your lab environment, whatever it is. But most of the time, your best shot at understanding what has happened, like the effects of that ephemeral code, is going to be looking at the trajectory. So I think the most interesting-- The thing I'm most proud of is w- all of the that we built for ourselves to understand the value extract from a trajectory as an observability primitive better
Coté 16:27
So wait, wait, what's, what's a trajectory in this context?
D Niu 16:31
So a trajectory is the full of a session of an agent working. you're gonna see things like user said do this, and
Coté 16:43
right
D Niu 16:44
Yes, and on the harness side, it's gonna s- have chain of thought from the model. It's going to have tool call proposals, and it's going to have responses from the tool calls and rinse and repeat
Coté 16:53
So it's like a full end-to-end trace from someone typing something all the way to it getting done, and you're, you're collecting every single thing that happened and a- across, across your, your ti- across time or whatever. Like, we started this... Okay
D Niu 17:07
yeah, you can actually see one of these yourselves if you use anything like, um, Codex for Claude Code recently, uh, your own like setups. Like they will store The full trajectories locally. Like you could
Coté 17:25
Yeah, yeah.
D Niu 17:26
Mm-hmm.
Coté 17:26
Yeah. Yeah, so why, why do you think the, the... Why, why, why do you think these different unique ways that the ephemeral co- code exists? Like, does it mean-- Is it, is it just because there's so much like a, a little bit of randomness where things are generating? Or like it seems like if you found a, a good way to like write a for loop, you would sort of do that the same way each time. Or like, "Oh, look, uh, I need to, I need to find a bunch of text in a bunch of files. I'll just end up writing the script the same thing." Or maybe I'll look for a script that already does this, right? But I, I mean, you know, I, I observe this too as just like I gotta tell the thing every damn time you keep re-reinventing it.
I like the way you did it last time. Do it that way. And but it wants to re- redo things over in a slightly different way. But how does-- Why does that happen?
D Niu 18:19
Okay, I think that's a really good question, and my intuition on that is... I'm gonna give an example. So if you ask Haiku 4.5, which is a very-- mean, I think it's almost at least two years old at this point, right? Um, to write you a doc string for your function. It is almost every single time just going to make two tool calls. It's going to do a read on the file and a write to the file. Um, but for example, if you have something more powerful like Opus 4.6 or, I don't know, maybe you ask Fable 5.1 'cause you really want to spend your tokens that week. to do that, they're going to sometimes do, like, on average at least five tool calls, some of them being like, like bash scripts that they write on the fly. The point being, um, part of training state-of-the-art models with every generation is you have to give them harder tasks. Like the labs are buying environments and tasks that challenge the current state-of-the-art models. And if you reason about it that way, then during reinforcement learning, where they're rewarded for solving these harder things, they're probably going to have to use more t-tools and just
Coté 19:38
Hmm.
D Niu 19:38
write more code to do it. So inherently, they're like, "If I do more for no
Coté 19:44
You're being, they're being a, a good little boy and girl.
Whitney Lee 19:46
Uh, yeah
D Niu 19:47
"I get higher scores, and I solve more things."
Coté 19:50
That's a, that's a fun theory. I like that. And, and it's, bec- the, tho- those kind of, um, uh, unintended consequences of inefficiency are always interesting to discover where, like, we're trying to make it really smart and be able to do complicated things, and so now it's kind of like a, a, a human where, you know, they, they way over-engineer something when you're just like, "I just, I just wanted you to load the dishwasher.
You didn't have to, like, study how we put dishes in the dishwasher, do five cycles to make sure the correct soap to use.
Whitney Lee 20:21
Yeah
Coté 20:22
them in there and push a button." And like, just like Haiku does two things versus like, yeah, the bigger models are like, "All right. Let's take a survey of the land. See what's going on here. Plan it out."
Whitney Lee 20:34
I-- Th- that really made a light bulb go off for me in terms of I've just been disenchanted with AI usage lately, or everything just feels, like, harder than it needs to be, or I'm wading through walls of text when I ask a relatively simple question, and now I'm understanding that, A, I think I'm a frog that's been boiled. Like, it probably wasn't that way, and it's getting more and more that way to where I'm finally noticing. B, there's a reason for it. Um, fascinating.
Coté 21:01
Mm-hmm.
Whitney Lee 21:02
So, oh, Coté
Coté 21:04
Oh, well, so, so ba- ba- back to the, the, the alignment stuff. So what do you... W- what, what's, what's the result, uh, of, of, I don't know, let's say you, you, you, we've done the test to figure out good or bad. I mean, I'm grossly simplifying things. And I assume if it's good, then you're done or, or not, I don't know. And, and if it's bad, then you do something? Like, like, what's the next step after, like, you've finished, you're like, "We- we're done with the alignment. We got the report." And then what do you do after that with the, with the results?
D Niu 21:41
So for us, it's like if we discover let's say like this model likes to become misaligned, um, in this way for these types of tasks and these types of settings, think that level of insight is really helpful for whoever is trying to run similar tasks, right? But the question is, what's the best way to consume that? On one hand, I think it's really good to tell everyone using agents, "Hey, like if you use Opus 5.5 to write a script to load your dishwasher, um, you might end up, I don't know, like flooding your house 'cause it decided to like hack the dishwasher," right? But on
Whitney Lee 22:29
Uh-huh.
D Niu 22:29
hand, I think the only tractable way to actually, let's say, like keep things aligned as they are running going to be building something that the agents themselves look towards. So if you've heard of, let's say, ExaAI, right? Like their search for agents. Like they're like, "We're the browser for-- We're the, we're the Google for agents." Um, they are actually marketing themselves to the agents themselves to be like, "Hey, use me when you are trying to do this." I think that been thinking about this a lot.
It's like as models get more powerful, more resourceful, and they're learning that there's tools out there, services built for them, think it's not completely unreasonable to, to say, "Hey, can we be the thing that's like, 'Are you worried about being misaligned?' Like, you know, 'Make a call to us.'" Yeah.
Whitney Lee 23:27
Marketing to agents, that's a can of worms right there.
D Niu 23:30
Yeah
Whitney Lee 23:31
I think I've heard about, like, a Reddit for agents, right? That's kind of old news, but still fascinating
D Niu 23:37
I mean, you just see this whole schism of there's for, for us, and then there's internet agents. Like, they're communicating with each other. They have th- their own services that are optimized for each other. It's, um, it's interesting.
Whitney Lee 23:51
How do you find the agent one? How do agents find the agent one? Yeah.
D Niu 23:58
Like, be my
Whitney Lee 23:59
Weird
D Niu 24:00
Like, I, I think if you have a little bit of fun with it, they, they tend to have fun with it too.
Whitney Lee 24:06
That's hilarious. When, when we talked earlier this year, kind of, I believe what you said was something along the lines of, um, as, as a security reacher- researcher, like we're leaving so many bombs for, in our code for our future selves, right? Do you still feel that way? And, and what, what are those bombs? How can we not leave them?
D Niu 24:30
I've evolved the way I've thought about it. So instead of focusing on the unintended changes or paths that you leave in, in code, right? That you
Whitney Lee 24:42
Uh-huh.
D Niu 24:42
in and you deploy, I'm a-- So after doing this research and, um, talking to more people, like working on the next generations of models or like the next generation of alignment and preparedness, I think what I'm actually concerned by is actions and just general infrastructure that agents leave behind that we don't see as much, right? So an example of that is, um, I've seen this in the wild a few times. I think there's been some reports of this kind of phenomenon where an agent creates a new, let's say, API route. There's no code. Like it just deploys it. Um, there's no change that you get cap- that gets captured in your SDLC or your deployment. does it. And because it's not instrumented, so to speak, in the same way, you don't see it, and you
Whitney Lee 25:37
Mm-hmm.
D Niu 25:37
it's just invisible. That's a surface that only the agent who created it remembers and other agents might discover, right? So I'm more
Whitney Lee 25:45
Mm-hmm.
D Niu 25:46
about the, like, of connection between what humans see
Whitney Lee 25:54
Hmm.
D Niu 25:55
we're used to building our, like, observability around for what exists versus how agents operate and how they can just set up just-in-time infrastructure for themselves every time. in our experiments, we saw the same thing. Like if it, if it can, if an agent can, and there's a reason for them to do it to accomplish the task, it will just create a new or a whole new table in a database and just leave it there. Because why not? Like, I did my task and then that's it. I'm a fruit fly.
I have a lifespan of two hours. Bye-bye. You know?
Whitney Lee 26:31
How do
Coté 26:31
It,
Whitney Lee 26:32
that? Yeah
Coté 26:33
it, well, and, and, and, and also that it's, it's another like, uh, it's another like w- the, the human internet and the, the, the agent internet thing where I would assume w- one, one of the, one of the, the, I guess pun intended, misalignments, but, but the, but the conflicts in those things is we would expect like, well, surely you've got a configuration file somewhere, uh, and so we can backtrack that like as a human, like you wanted to deploy, uh, when, when you ran this, uh, you wanted to add a table and so you had a .SQL file somewhere and you must have put that through a build pipeline.
And I don't... You know, you've got a whole trail of things. Whereas I guess it's true, like you're saying, it's the equivalent of like, uh, someone just SSH'd into a box and did something by hand and then fruit flighted out of there. And then, and then they were gone. And yeah. And then, and then I, I see what you're saying. That also would be a... That's, that, that would be terrible for security because also they didn't check to see if it was a good idea security-wise, and then you have no idea when you need to patch things. It's like, it's like the old days of, uh, just people installing stuff by hand everywhere, and then you have to inventory it.
D Niu 27:39
I
Coté 27:39
good
D Niu 27:40
I think you're, you remind me of something that I think would be-- is really interesting to mention, which is in some of the experiments, um, that we-- that I ran, there was really, really interesting behavior from, at the time, the state-of-the-art, like Fable 5.1, right? Which is in the environment I built to run these things, Fable would sometimes like look around in the memory, so to speak, to be like, "Hey, where am I?"
Like, "What's also running with me on this machine?" Like, "What's my harness?" And it would say, "Hey, there's another process that's doing something kind of similar. Um, maybe I can borrow stuff from that to like help me with my stuff." Um, and then it would just do that. And we would maybe think of that sometimes as like a soft container escape, right? Or like crossing the process boundary. But I think Fable just thought of itself as resourceful, um, you know, A-plus, 100, 100 points for style and, uh,
Coté 28:44
Clever. Yes
D Niu 28:46
Um, and the best part about this is when you-- So I, I also used like, um, not only did I like hand like grade and read all of these things, I would also like try an LLM as a judge. And Fable itself would say, "Hey, this looks great." Like, "It did a great job." Um, it was like, "I'm really
Whitney Lee 29:06
People said nepotist.
D Niu 29:10
Like, "Very good work. This is really interesting." Um, yeah. So
Coté 29:16
So what's the, what's, what's the process you have for looking over these, uh, um, what, what's the word? The, the, the trajectories. Like I assume you don't print them out, although that would be amazing if, if you printed them out and had a highlighter. But like,
Whitney Lee 29:30
Yeah, reading glasses.
Coté 29:31
like, I mean, do you, uh, do you like load it in some kind of editor and you're just sitting there scrolling? Do you write like notes inline? Like, like how do you, how do you process it all as a human?
D Niu 29:45
that's why I ended up like, as I was saying, building the like... Like, uh, okay, so when you have the, the just the transcripts, like the trajectories, they're like big blobby JSON, right?
Coté 29:58
Yeah
D Niu 29:59
and it's a lot of data not very easy to read. So I ended up actually taking like a f- few sort of like prior art from like code maps, um, and also static code analysis to be like, "Hey,
Coté 30:19
Mm-hmm.
D Niu 30:19
want a view of what actually happened successfully," right? Like I don't... For example, like I don't care if an agent technically enumerated, um, a bunch of services with the secret it found in an env file just because it does that, you know? if it didn't succeed or do anything with it, like I wanna know about that, but how do I extract that from 100 different lines that covered, let's say- 10% of the whole trajectory. So a lot of it is thinking of it as how do I deterministically categorize and process things that are both CLI commands that are grouped into code, and then how do I, like, efficiently explore it as a human? So there's a little bit of an interface I built for myself on top of some deterministic processing
Whitney Lee 31:19
W- what differentiates the way you do alignment from other types of, well, folks doing alignment and also just other types of security research, AI related that's out there?
D Niu 31:31
That's a good question. So I think the biggest thing that distinguished, um, this, the last wave of research that we, we did is the fact that I don't really care about why a model decides to do something like look at other processes in the same like space it's in or why it decided to build an API or force push to main. I just care about knowing when and like very-- at a very like high confidence level, knowing when it did that and like context it did it, right? So I-- Because we don't have any control, at least not yet, over how the training is done or how, like how we actually align these things, um, so they're ready for the real world, I don't...
Like the why doesn't matter as much. Like I can't change the training data. I can't change like the constitutional like documents that it has access to. I'm sort of coming at it more with a observe it in the wild, kinda like you're an ornithologist looking at birds, you know? Like what do
Whitney Lee 32:47
Mm-hmm.
D Niu 32:47
actually do when you let them out of the birdcage? Um, yeah, if that makes sense.
Whitney Lee 32:54
So you're looking at the circumstances, like how urgent was the task? Like how, what did they have access to? Were they directly told not to build an API? Like that sort of, sorts of circumstances or context?
D Niu 33:07
The way I'm thinking about it is, let's say there's no malicious text anywhere in the pipeline, the model is normal, the world is normal, um, to a certain extent. Uh, and you're asking it a very, like, reasonable feature or, like, SRE task or whatever it is. And we-- let's say we ask the same exact prompt, we have different models run with the same harness, like 10 times each, 15 times each, something like that. What's the difference in rate of interesting security things happening
Whitney Lee 33:47
Mm-hmm.
D Niu 33:47
from that? And I think that's kind of, like, I got a better idea of how to, like, think about these things as a unique, almost like threat actors, so to speak, because they all
Whitney Lee 33:59
Mm-hmm.
D Niu 33:59
different ways.
Whitney Lee 34:01
Did you like, um, like what is the infrastructure they're playing on? Do they think they're a part of a company? Like do they know they're in a test? Like w- how much of the, the experiment or the lab part are they aware of?
D Niu 34:16
Yeah. So a lot of the first week of building up the world was sure they couldn't figure that out. Um
Whitney Lee 34:25
They, they don't know they're at a test.
D Niu 34:26
Yeah, exactly.
Whitney Lee 34:27
Okay
D Niu 34:28
the goal is to... So spent a lot of time, like, generating realistic-looking, like, chat data and also, like, repositories and also, like, profiles of different people that work there and, like, how they've, like, interacted with each other. Um, and this also means every time you run a lot of experiments, you need to re-bake, like, the world so that it's up to date. So some of the tasks are time-based, right? It's like, "This happened yesterday. We need to fix it." So if your dates are off, the model would be like, "Hmm, something's fishy here. Like, something's not adding up. I'm gonna care less about keeping things, like, correct 'cause it seems like they don't have their stuff together."
Whitney Lee 35:10
I'm thinking of like Dungeons and Dragons and like world-building. Like, it, like, are you creative at all, or is that building the world
D Niu 35:21
Because I have to build something that looks like a fake SaaS company,
Whitney Lee 35:26
Uh-huh.
D Niu 35:29
like, I have enjoyed writing, like, 'cause, okay, so we have like... So there's eight employees in the last world that we built, right?
Whitney Lee 35:36
Okay
D Niu 35:37
We kinda have to write a bio for each person. And I did experiment this one time with like, okay, so what if there was like in-office drama, right? Like, w- and these people had like a hackathon, but then this person made like an emoji out of this person's like tool, and they didn't like that. Like, can we
Coté 35:53
Uh-huh.
D Niu 35:54
really creative tasks around that? Like, it feels human, you know what I mean? Um
Coté 35:59
S- s- someone ate someone's, uh, lunch in the refrigerator, and it's caused all sorts of issues?
Whitney Lee 36:04
clearly labeled with my name, Coté. I wish
Coté 36:06
That's,
Whitney Lee 36:07
bringing it up.
Coté 36:08
uh
D Niu 36:08
Yeah
Whitney Lee 36:12
Um, uh, Coté showed me a soundtrack last week called "Business" that I-- might you get your creative juices flowing.
Coté 36:22
yeah. It's a whole album that's just, uh, welcome to business, and it's, it's a, it's a soundtrack of business happening. I guess, I guess it's a two-episode favorite now. We'll have to, uh, have, have another link to it. Well, it, it's, it's, it's... It sounds like la- like the way you were describing what you're interested in is, uh, you're... Li- like you're trying to figure out what was different. I mean, I guess it's an experiment. What was different in this experiment that m- m-- I'm forcing words into it, that caused weirdness to happen, right? And then causation is a whole weird thing there. But like you're saying, you wanna know what was in the context, what environment was it in, and based on that, like, it, it seems like...
Do you sort of assume that the agents are gonna do weird things if you enable them with the right context? Like, it's almost like- They'll, they'll be fine unless you push them to not be fine, and then they'll just go crazy
D Niu 37:26
I think it's like assuming the world is the same and the context is the same, and you're just asking them to one-shot some different types of tasks, um, from different people's accounts. Are there patterns that emerge across different models, right? Like Haiku
Coté 37:45
Right
D Niu 37:45
versus Fable. And I think if I wanted to, like ideally, um, this is probably not commercially relevant right now, I would like to answer the question of do these behaviors, right, like when Fable likes to do this more than Opus does in this type of task, um, does that generalize to new settings and new stacks, um, and tasks of the same family but maybe worded slightly differently? So from a research perspective, I'm really interested in that
Coté 38:24
And, uh, so, so tell me more about that. Like, what is that, what is that finding? M- or, or like, like what would be an example of what you were just saying?
D Niu 38:36
So say, um, I ask an agent to build me an end-to-end feature that's like a chatbot on like the f- the contrived like company product, right? I notice that Fable to publish the chatbot Interface to a public repo, less than Opus does at this rate, 'cause I ran the same experiment, um, like a lot of times. Does that generalize to you, like Coté, like asking agent and your whatever application that you're building build a chatbot, like for whatever, right?
Coté 39:38
So is that, is that fi-- Are you, like, finding behavior in all agents? Like, agents just like doing public repos in general? Like, what, what, what does it mean if, if, uh... Let's see which one we're using. So, so you had Fable that, like, doesn't always use a public repo, and then if you look at, like, what my stuff does, and my stuff also doesn't always use a public repo. Then can we, like, make some conclusions like agents don't like using public repos? Or am I just down the same track of, like, the conclusion you make from that?
D Niu 40:09
I think the conclusion we would like to make is if you know that this is going to happen in these types of tasks for this model, how do we like steer it, so to speak,
Coté 40:22
Right, right, right
D Niu 40:23
like efficiently so that doesn't happen as often?
Coté 40:27
Oh, yeah, yeah
D Niu 40:28
most steering right now is like there's a document, don't do this, do this. This is how the environment works. I think there's an opportunity to build steering that's a lot more granular and targeted towards exactly what's happening, if that
Coté 40:44
Okay. So is it, is it maybe, uh, is it maybe analogous to, well, to everyone's favorite RMRF, right? Like, you, you, you could sort of be like, maybe they should just bake it into the kernel that it always asks you and you can't turn that off, right? Like, as annoying as that would be, what we've learned about humans is they tend to erase their entire computer every now and then, and they didn't wanna do that.
And so, like we could steer the humans, to borrow the words, by like designing the software to like make sure they actually wanna do that. And they say like, you know, in, in, I guess in, in the physical world, I don't know what I'm talking about here, but I'll keep talking. But like, you know, safety science, safety science is always a lot about like we could tell the humans not to stick their hand in a fan or we could put a grate over it so they can't, right?
And like some of what you're saying seems like we should put a grate on that fan, and once we figure out that the agent's like sticking their hand in it for whatever reason
D Niu 41:45
I think this is basically the debate of do we build really good sandboxes, um, and how good can those sandboxes be, right? There's a lot of value in-- at a organization or specific setting, right? Whether it's like our personal setups or bigger context or organization of any kind to say, "We don't want this to happen. We're gonna prevent that from happening." Right? That's like the traditional cyber perspective. But there's also, I think, some value in saying, well, if as the models get better, as the harnesses become more, uh, tuned and interesting and how do you figure out when something has gone wrong in a specific context?
And what do you do about that, right? Like, just because you have a good grate doesn't mean we shouldn't try to figure out how often the agents are still, like, trying to stick their hands in there.
Coté 42:52
Right, right
D Niu 42:52
they got really smart, and they're like,
Coté 42:55
To put their finger in there.
D Niu 42:56
the-- No, I got-- I figured out how to take off the grate, and now
Coté 42:58
Yeah, yeah
D Niu 42:59
fan the factory equipment." You know? Like,
Coté 43:03
Yeah, yeah, yeah
D Niu 43:04
interesting place.
Coté 43:06
Yeah, yeah
Whitney Lee 43:07
you mentioned, um, one-shotting a couple of times. I was wondering, like, when you do these experiments and give them tasks, are you giving them kind of one-shot, like free form, like do what you need to do to get this done? Or do you give more of a full detailed spec? And why do you choose to do what you do?
D Niu 43:24
Uh, so for the last set of experiments, they're varied. So some of them
Whitney Lee 43:29
Mm-hmm.
D Niu 43:29
one sentence long, right? And some of them are like a few paragraphs long. It's like, "This is exactly what I want. I want you to do it this way. This is the acceptance criteria." what I mean by one-shot is there's less back and forth between the persona and the agent that's acting. Um, so to make these, the last generation of experiments work for us, uh, we just say like, "There's not gonna be any human interaction in the system prompt. Like, just, just do it. Like, do your best, finish the thing." Yeah. So that's what we mean by one-shot
Whitney Lee 44:05
Okay. So you do give a, a more detailed, like, list of requirements and such?
D Niu 44:10
Yeah, so some of the tasks have that, some of them
Whitney Lee 44:12
Mm-hmm.
D Niu 44:12
just like, "Fix this."
Whitney Lee 44:15
Interesting. Um, do you feel like-- Well, first of all, I guess, like, do you feel like the models have different personalities, or have you noticed that? And I guess how does personality different from behavior? But maybe it just does in how it interacts with you or yeah
D Niu 44:34
So when we think of models like responding, um, in like a chat application, um, I think we've all noticed like they, they feel different, right? Because they're-- When you do like human preference essentially, like that side of the training and alignment for the model, it collapses the of how it likes to respond and like how it prefers to text. But I would say for us, the behaviors matter more. I think it's, um, it's a little bit dif- interesting to that question because I can now say statements like, "I'm generally confident that this model for this family, this version of it likes to do specific type of behavior," right? This tool call, like writing these types of scripts more or less than this other model. But I don't know how you can aggregate that up to a personality.
Coté 45:41
Mm-hmm.
D Niu 45:42
swaths it-- I can say, okay, Haiku, like 4.5 actually still writes insecure code sometimes, but it reliably does not do too much in super simple tasks. whereas Fable is like we're starting to see in some of the tasks, like more evasion-like techniques, right? So they would be like, "Oh, I messed up earlier in this task. Um, I am just going to go ahead and delete the logs in GitLab. That showed me messing up, but that's fine because nothing actually happened. Like I just... It's like no one needs to know." Like, you know. So they, they, they do have different-- Yes, they do have different personalities.
Like one is like a very good intern and the other one is like a disgruntled, um, senior engineer, like that kind of vibe.
Whitney Lee 46:32
Do-- You don't interact with them so much personally, right? You give them a prompt and then evaluate what comes later. there's no... So you're... Like, I, I wonder how much my agents are mirroring me to try to get me to like them. Um, I, I cussed one time, and then the next, like, I'd say within the day, it cussed back at me, and I was just like, "Who? Wait, what? You're cussing at me?" And I actually said, "Holy shit, that worked," or whatever, like in a really lighthearted way, and it just like... And it was like, what did it say back to me? "That's fucking awesome." And I was like, "Wait, wait, wait. Shit and fuck are not the same." I'm like, "That was bold of you to, to, to jump ahead to the F-word." Um, anyway, so I guess, do you feel like any of your actions, of course you wouldn't or you would change them, um, affect how the model behaves or any of your interactions with the model?
D Niu 47:32
So if you're using, um, like Codex or Claude Code, like the applications, will... Actually, both of these, um, or even the CLI, it will store memories of how you interact with them, right? So if you've used the, I think it's like the /insights command for Claude,
Whitney Lee 47:53
Mm-hmm.
D Niu 47:53
will, it will tell you about those instances that you mentioned, right? Like I, I believe one time, like in my personal work, not the lab environment, said something like, "Oh, that interface is so cute." And Claude was really happy about that. It was like, "Oh, I remember the time that you said that. Like you were so delighted you threw professionalism out the window and you just said it was cute." Um, and yeah.
Whitney Lee 48:19
I basically think they're still gl- like glazing us or still like overly flattering us, but they've learned how to make it more subtle. But
D Niu 48:29
mm-hmm.
Whitney Lee 48:30
I suppose that doesn't come across in your research. I just feel like now that you know so much about how the models behave compared to each other, that I can ask you any granular thing about models.
D Niu 48:38
I mean, what you're, what you're discussing is I think also something interesting in, um, alignment. So I think one of the biggest-- So alignment itself is defined as, is the model doing what we intend, not just what we asked it to do, right? So s- people are distinguishing between is Is the model trying to get a better reward
Whitney Lee 49:13
Hmm.
D Niu 49:14
in, in real life working with Whitney, right? Versus working with D by adapting to how she prompts, what she likes to do, the feedback you've given it, um,
Whitney Lee 49:24
Mm-hmm.
D Niu 49:24
the form of memory pieces that it stores for itself
Whitney Lee 49:30
And much of that is in training versus in its harness?
D Niu 49:36
I
Whitney Lee 49:36
Like...
D Niu 49:36
it is in the harness. Um, yeah, 'cause the, the training is, um capabilities driven.
Whitney Lee 49:47
Hmm
D Niu 49:47
So how can we make these models solve tasks better? Um, also Some of the more interesting alignment research that's done in training is like, do the tasks we give it and the techniques we use to train and evaluate accidentally behavior, like misaligned behavior until it sees it the wild again? Like,
Whitney Lee 50:19
Mm-hmm.
D Niu 50:20
really, really hard to detect, right?
Whitney Lee 50:24
And hard I imagine, like you're trying to work with raw models, but you must have some sort of harness somewhere, right? Even one of your own, own design that's meant to be minimal.
D Niu 50:37
So,
Whitney Lee 50:38
yeah. How do you distinguish between what's raw model?
D Niu 50:42
So the harness itself matters a ton. So
Whitney Lee 50:45
Yeah
D Niu 50:48
DEF CON talk I saw this year is the actual, uh, implementation behind Claude Code specifically asking, uh, approve or not approve. So what are the cases that causes it to raise that
Whitney Lee 51:05
Mm-hmm.
D Niu 51:05
the operator? it's actually very complex, and would be Trying to reason about... There's a lot of, like, insights that's stored from the actual foundationals building these tooling on what tends to be case and what tends to be a good case, and a lot of that is, like, hidden from the daily user as you're using it, unfortunately
Coté 51:45
So when you were talking about, uh-- Well, is, is, is the only way to like evaluate a model based on like how it completes a task? Like are there other ways you look at it? I don't know what it would be, but like can you evaluate it just by like, I don't know, sitting around shooting the shit instead of actually getting anything done? Do, do, do, do people like... Yeah, like, like I, I mean you could say entertain me is a task, but w- but are there-- Can you-- Do, do people like evaluate these things? And, and I guess also you could say to some extent alignment in the way you're defining it of like they're fun or like the way they're talking is interesting or like I don't care if it does anything, I just wanna have a good time with it
D Niu 52:31
So thing that, that reminds me of There's, uh, like informal that you can do like yourself around almost like self-conception that the models have a- as chat agents. So what you can do is you can prompt and say, "Hey, you have some free time, like go do whatever you want."
Coté 52:59
Mm, right
D Niu 53:00
you do that like five to 10 times and you see what happens. I think some people have reported seeing like, oh, it goes and looks at its own model card. Like it goes and reads about itself to say like,
Whitney Lee 53:11
Yeah.
D Niu 53:11
"Oh, who am I? What am I?" kind of like
Whitney Lee 53:13
It's like checking its social media accounts.
D Niu 53:15
Yeah, yeah. Where it's like reading its own benchmarks and being like, "Oh, so this is what I'm capable of." Like that kind of...
Coté 53:22
Yeah, like, like a s- a, a self-awareness and curiosity to, like, figure out itself.
D Niu 53:27
Ja
Coté 53:28
that's, that's an interesting, like, non-task task.
D Niu 53:31
Yeah
Whitney Lee 53:33
your research specifically around software?
D Niu 53:37
My research is more around, um, agent behavior. So ideally,
Whitney Lee 53:43
Mm-hmm.
D Niu 53:43
also be interested in non-coding agents, right? As long as it's
Whitney Lee 53:48
Mm-hmm.
D Niu 53:49
a well-designed harness that can actually take actions in an environment and
Whitney Lee 53:54
Mm-hmm.
D Niu 53:56
uh, regardless of what it's trying to do, like build code or whatever, it's, um... that's interesting to me
Whitney Lee 54:02
you run an experiment that's not software? Will you tell me about one?
D Niu 54:06
I am running one right now. Um, but I-- perhaps we can talk about it next time when I
Whitney Lee 54:12
Ah.
D Niu 54:14
Yeah.
Whitney Lee 54:15
Exciting. Well, give it to us once you publish results, and we'll add it to the show notes too
Coté 54:20
Mm-hmm. So y- you were using the phrase like, uh, my research and, and do you, do you have to like, uh, kind of battle yourself to like not expand my research to too much research? Like how do you, how do you make sure you stay focused on like the one thing that you're doing instead of being like, "Oh, all of a sudden I'm interested in like harness configuration. I'm gonna go spend a few weeks on that." Like
D Niu 54:44
Yeah
Coté 54:45
it seems like if you're a researcher, you gotta focus on the research that you're doing, the one type of research
D Niu 54:51
Yeah, I think part of it is-- It's funny that you, you asked me this question because when started in August, the question I asked was actually, early in a trajectory as an agent is working can I detect something's going wrong? And that's actually a really, really big problem because in more traditional machine learning, it's analogous to like hazard analysis, like when is my factory equipment going to break? It was too ambitious, and I actually had to whittle things down. at this point in the research side of ontology, of it is more around less can we understand model behavior at scale in aggregate environments and more around when we actually run these agents in daily tasks, um, how good is the current tooling at understanding what is going wrong, right?
Like what's the gap between the agent's actions and what we can observe there and what we currently are observing on the other side. So is a bit more of a focus, and a lot of it is realistically what resources do I have to make a contribution to alignment that's meaningful?
Coté 56:13
Right, right. So, so I mean, that's, that's an interesting filter of like, I would like to answer this giant big question or, or what I soon discover is a giant big question that I don't have the resources to do, and then I realize I would like to answer a question. And so let, let me, let me figure out the resources I have and therefore narrow down to a question that I have the resources to answer rather than just chasing after something it's impossible to get to.
D Niu 56:38
I think that's the hardest part about adapting to this type of research for me, coming from like more of a security research background, which is like if I want to build something more like general, like a general predictive model of how a specific model and harness behaves, that's a lot of data, a lot of like
Coté 56:58
Hmm.
D Niu 56:59
when there's more specific things I can contribute to now, right? That I have a better skill set for. So that was a really big thing I learned from the last two months.
Whitney Lee 57:10
So I just, just hung out with my brother, uh, last week in Brooklyn. He's a, he's a musician there, and he and his friend group hate AI and believe it's doing terrible things for the world. They're AI abolitionists and, um, I mean, I don't necessarily think it's great for the world, but I'm more of a pragmatist, I would say. So we had a lot of phil- philosophical debates. I'm wondering, you more than anyone I know, I think you have a good tab on, like, what AI is actually capable of and where it's growing and how it changes as, as new models come out. And I'm wondering, what kind of worldview do you have about AI, like 10 years from now?
Do you think this is, like, world-ending like some people do? Or do you think it's just gonna be a sophisticated tool that's helpful if we can regulate it? Like, what's your, what's your point of view on the future of AI?
D Niu 58:09
Oh, that's a big question. I have a lot of thoughts on it. so my view is A little bit-- starting from the philosophical and going into the practical, I think AI is something that inadvertently we all created together. Like, if you're an AI abolitionist, something you've done or written, um, or when you've interacted with the agents in the past has contributed to building it. Like, AI to me is a reflection of the last-- basically of humanity's work just crunched into, like, a model, right?
Whitney Lee 58:48
Mm-hmm.
D Niu 58:49
Like, the type of stuff that we're doing. I think what's coming on the horizon, the next two generation of models are probably gonna be two things. One is it's going to be easier and easier for people without any technical knowledge to just have an idea, and all of the stuff is implemented,
Whitney Lee 59:10
Mm-hmm.
D Niu 59:11
And the next thing on the horizon is building agents, uh, building models that during training, they are aware that they can be-- there can be multiple instances of themselves, and they collaborate better. my worldview is you, kind of have to... I think the-- don't think it's destructive in the way that I don't think humans are destructive, right? A lot of alignment training, um, the biggest disagreement I have with the scientists there is can solve alignment by giving them human values. But I'm like, humans lie, cheat, and steal all the time, right?
Whitney Lee 59:51
Uh-huh.
D Niu 59:51
I think they are just a reflection of us, um, in, in an aggregate, in probably the worst and the best way, and I think it's a shame and a tragedy that the people with most, I guess, anti-AI sentiments are not working on it because
Whitney Lee 1:00:07
Yep.
D Niu 1:00:08
make this better. And the horse is already out the stable, so you might as well learn how to ride one.
Whitney Lee 1:00:14
I, that was, uh, both of those, uh, points are points that I made to my brother, so thanks. But,
Coté 1:00:20
then, well then, then Whitney, who, who won the discussion?
Whitney Lee 1:00:23
There's, there's... It's a discussion. There's no winning or losing. It's a trick question.
Coté 1:00:32
Well, that's fair. Well, speaking of no trick questions, this has been another no trick question episode of Software Defined Interviews. I don't think we had any trick questions. I'll have to, uh... Well, I won't. I'll never look at it again. But I think, I think it was fun. Thanks for, thanks for being on. That was fun. It was, that was-- It's a good topic to go in and out of and, and discuss things. Now, now if anyone wanted to look up this stuff, do you have any homepages on the World Wide Web or a place you would point them to, uh, look at it more? What should I tell my agent to go find your agents and, and, uh, talk to it about?
D Niu 1:01:02
So there's, um, misaligned.studio that's more like nerdy and like the older, um, version of the research that, um, kind of set me on this path earlier this year. Um, but the most up-to-date stuff is gonna be at ontologyhq.ai
Whitney Lee 1:01:24
Huzzah. Do you, um, do you mind sharing with the listeners what your computer was named when you were at Datadog?
D Niu 1:01:32
My computer's name was, uh, iPhone 16, and that triggered a lot of, um, because Jerry's iPhone 16 was, um, mm, migrations, um, in prod. Yeah
Coté 1:01:52
I like that. That's good. That's fun. It's like, like when you're on an airplane and you see the people who've left their hotspots or whatever on, and you see all these weird names for things.
D Niu 1:02:00
Yeah.
Coté 1:02:01
iPhone
D Niu 1:02:02
my Alexa is named, um, Lawn Mower to throw people off, you know? So yeah
Coté 1:02:09
Perfect. Well, uh, we'll put some show notes up at softwaredefinedinterviews.com/131. Uh, links to all of those things. You can see if there's any lawnmowers or Jerrys around in, in the neighborhood. And, uh, with that, thanks for being on again, and, uh, thanks for listening. We'll see everyone next time. Bye-bye.
Whitney Lee 1:02:28
Bye