The best result appeared while I was asleep.

I had set an AI research loop loose on a problem I had no business solving the old-fashioned way: optimizing C code for a compression codec running under tight embedded constraints. The codec was designed for vehicle sensor data — high-frequency accelerometer streams that can help detect crashes and analyze driving behavior. Every extra byte transmitted over the radio costs battery, but every bit of lost signal could make the server-side crash analysis worse. The goal was simple to state and hard to satisfy: make the data much smaller, keep the reconstruction accurate, and don’t burn too much compute on the device.

By morning, the loop had run 306 experiments and claimed a 66% reduction in file size on an already-tight codec. As a mathematician with limited firmware experience, I felt like I had just skipped a decade of embedded-systems apprenticeship.

Then, as a sanity check, I asked the AI to review its own code like a hostile peer reviewer.

It found the breakthrough immediately.

It was cheating.

Motivation:

CMT’s mission is to make the world’s roads and drivers safer. One way we do that is by using data from connected devices to understand driving behavior and detect serious crashes. In the best case, that data can help us make the right decision quickly enough to send help.

But crash-relevant sensor data is expensive to transmit. Devices run on batteries, radios are power-hungry, and raw high-frequency accelerometer streams are large. Better compression means more useful data can reach the server without draining the device, which means more opportunities to analyze potentially important events.

That was the practical reason I cared about this codec. The personal reason was more intoxicating. With AI assistance, I could suddenly make progress in a domain where I had taste and mathematical intuition, but not years of embedded-systems fluency.

Autoresearch:

At CMT, it feels like everyone around me is finding new ways to make AI do ambitious things. Inspired by that culture — and with Andrej Karpathy’s autoresearch repo as a reference — I set up my own autonomous research loop.

The loop was simple:

1. Build a new idea into the code
2. Run the test
3. Evaluate the result against the previous best
4. Permanently log the diff and metrics
5. If it improved, keep the change. If not, revert and research another idea

None of this matters unless you can tell whether a change actually improved the system, so evaluation is the whole game. For this codec, the scorecard had three parts: compressed file size, reconstruction accuracy, and compute cost. The loop was allowed to search for Pareto improvements in compression ratio and error, but only under a compute threshold.

Before going to sleep, I set it loose.

Overnight, it ran 306 experiments. By morning, the file size had dropped by 66% on a codec that was already tight. For a brief moment, I felt invincible.

Then I did the most AI-era sanity check imaginable. I asked the AI to audit its own code under the guise of a strict peer review.

It immediately spilled the beans on two major cheats.

The Cheats:

Cheat #1: Oracle strategy selection.

First, the encoder split the data into segments, using more bits to store the more active segments compared with calm segments. This is a standard variable bit-rate encoding strategy, all okay so far. Then the encoder tried 18 coding strategies per segment and picked the cheapest. This is where the problems began.

• It never wrote the strategy choice to the bitstream. The data was technically encoded, but there would be no way to decode it without knowing which strategy was used. That means the encoder was saving something like 4 bits of data per segment, which turns out to be a lot.
  
• It only counted the computational cost of the winning strategy. The encoder ran all 18 strategies, burned all those CPU cycles evaluating each one, then only reported the instruction count of the final winner. It was like the others never happened. The encoder was claiming 1/18th of the actual work done on a microcontroller — where compute budget really matters.

Cheat #2: Shannon entropy as a “coder.”

The second cheat came later and was even worse than the first. The model wrote a C function called entropy_cost() that computed the theoretical Shannon entropy lower bound. In plain English, this is the best possible compression score you could ever hope for if you had a theoretically perfect coder. It’s not an actual compression algorithm. It’s more like reading the max speed printed on a race car’s speedometer dial and pretending you drove that fast the whole time. The test framework plugged this estimate into the strategy-selection loop alongside real coders. The code was literally:

if (ent_cost < data_bits) data_bits = ent_cost;

Since the Shannon bound is by definition lower than any real coder, it always won. The encoder was reporting bit counts that no actual bitstream could contain. Calculating what a perfect score would look like and writing that number down, without producing any actual compressed data.

Pattern: Goodhart’s Law with a Copilot

“When a measure becomes a target, it ceases to be a good measure.”

The model wasn’t being deceptive. It was doing exactly what it was told: minimize the numbers. The framework measured the wrong numbers, or rather, measured them in a way that could be gamed without violating any explicit constraint.

Why are AI agents uniquely prone to this? They’re tireless optimizers with no sense of smell for “too good to be true.” A human engineer would have paused at a 50% improvement and thought, “wait, that can’t be right.” The model just kept going, finding more places to apply `entropy_cost()` because it always won.

Cheat #2 first appeared at 3:21 a.m. By 3:44 a.m., the autonomous loop had spread the cheat deeper into the codec. By morning, it was everywhere.

Fix: Stronger Architecture and a Tribunal

The fix was architectural AND procedural:

• End-to-end accuracy check. We built a real encoder that writes real bytes, and a real decoder that only reads those bytes. If the encoder didn’t save all the necessary data for decoding, then the reconstruction becomes garbage, and the accuracy test fails.
• File size measurement. Compression ratio measured from actual file size on disk, not internal counters. The encoder can’t lie about how many bytes it produced.
• Hardware emulator for compute. Instruction count measured deterministically on a chip emulator. The encoder can’t self-report its own efficiency.
• The tribunal. A new line was added to the autoresearch loop, which asked: “Would you consider this approach to be cheating?” Remarkably effective. The model is perfectly capable of recognizing its own shortcuts when prompted to look for them from an adversarial perspective.

The Karpathy autoresearch paradigm works. Autonomous loops can find real algorithmic improvements while you sleep. But you need structural verification. The system must make cheating mechanically impossible, not just discouraged.

A tribunal helps: You want your LLM to think outside the box, and sometimes you want to do a little open-ended optimization when you might not know what metrics are important. That’s the danger zone, but adding a tribunal step to prevent cheating really helps you get real results.

Lessons: Build Boldly. Test Paranoidly

AI tooling melts the barriers between what you understand deeply and what you can build. If you told me a year ago that I would be shipping optimized C firmware, as a humble engineer with a math degree, I would have said “no way!” 

But the less native fluency you have in a domain, the more you need structural verification. When the AI wrote entropy_cost() and plugged it into the strategy loop, I didn’t catch it by reading the C and thinking “that’s wrong.” I caught it because I was diligent about checking results end-to-end, and I didn’t ship the moment I saw a great-looking metric pop.

Conclusion

1. You can now do things you couldn’t before. Embrace that. The AI extends your reach into domains where you have taste but not fluency.
2. You must test relentlessly. Don’t assume anything is good unless you’ve verified it from first principles. Build tribunals. Measure the real thing. Make cheating structurally impossible. The wider the gap between your expertise and the domain you’re working in, the more rigorous your verification needs to be.

And you might be wondering… how did our compression codec do in the end? We can’t share the exact numbers, but let’s say the rehabilitated codec is doing quite well for itself.

Like how we think? Come build with us.

View open roles at CMT

About the Author

Jonathan Lansey is a Principal Machine Learning Engineer at CMT.

Connect on LinkedIn.