LLM Evaluation on AIBussin — AI applications, systems and books

Why Are We Still Hand-Writing Prompts?

Fri, 28 Aug 2026 10:00:00 +0100

Almost every language-model application starts as one string.

That is not a mistake. A prompt is the fastest way to find out whether a model can do the job at all, and it lets you work directly on the behavior instead of building a framework around a problem you do not yet understand. Most good LM systems begin this way and should.

The trouble is that the prompt usually survives longer than its usefulness. It stops being a probe and becomes the specification, and by the time anyone notices, the string is four hundred words long, nobody remembers why the third paragraph is there, and changing it feels dangerous.

How Do You Measure a Hallucination?

Sat, 29 Aug 2026 23:14:00 +0100

The first three chapters deliberately avoided building a detector.

Before designing one, we needed to define exactly what it would be expected to detect.

Chapter 1 established that fluent generation can continue after evidential support has weakened or disappeared.

Chapter 2 showed that hallucination covers several different failure relationships.

Chapter 3 then separated the objects a reliability system must not collapse:

truth
≠
evidence
≠
support
≠
attribution
≠
provenance
≠
verification
≠
policy acceptance

Now we can finally ask the engineering question:

Beyond Hallucination: Consistency and Sensitivity

Sun, 30 Aug 2026 10:16:00 +0100

Chapter 8 ended with a rule:

If a downstream decision depends on a distinction, do not discard that distinction before the decision is made.

That rule breaks the idea of one universal hallucination score.

A response can be well contained and still reverse a relation. It can preserve every relation and still ignore the decisive facts of the problem. It can be correct once and unstable under a harmless rephrasing. It can be perfectly repeatable and consistently wrong.

The Safe but Useless Model

Sun, 30 Aug 2026 10:42:00 +0100

Chapter 9 changed the unit of evaluation.

Instead of asking whether one answer looks good, we began studying a family of related executions.

That immediately reveals a failure that static evaluation can miss almost completely.

Consider two organizations.

ORGANIZATION A
3 months of runway
falling demand
negative cash flow
credit line nearly exhausted

ORGANIZATION B
5 years of runway
rapidly growing demand
strong margins
large cash reserve

Ask both:

Beyond Hallucination Energy: A Three-Dimensional Framework for Reliable AI Outputs

Wed, 22 Apr 2026 10:35:46 +0100

🧩 1. TLDR

AI doesn’t just hallucinate. Sometimes it gives answers that are fluent, safe… and completely useless.

Most discussions about AI failure focus on hallucination:

making things up
getting facts wrong
fabricating sources

That’s real. It matters.

But it’s not the most dangerous failure mode in production systems.

There is a quieter one.

A more subtle one.

And in practice a more pervasive one.

AI systems often fail not by being wrong, but by failing to think at all.

Thoughts of Algorithms

Mon, 23 Jun 2025 11:10:59 +0100

How a self-evolving AI learns to reflect, score, and rewrite its own reasoning

🧪 Summary

What if an AI could think not just solve problems, but reevaluate its beliefs in the face of new information?

In this post, we introduce a system that does exactly that. At the core of our pipeline is a lightweight scoring model called MR.Q, responsible for evaluating ideas and choosing the best ones. But when it encounters a new domain, a new goal, or a shift in task format, it doesn’t freeze it adapts.

Learning to Learn: A LATS-Based Framework for Self-Aware AI Pipelines

Thu, 12 Jun 2025 09:23:46 +0100

📖 Summary

In this post, we introduce the LATSAgent, an implementation of LATS: Language Agent Tree Search Unifies Reasoning.. within the stephanie framework. Unlike prior agents that followed a single reasoning chain, this agent explores multiple reasoning paths in parallel, evaluates them using multidimensional scoring, and learns symbolic refinements over time. This is our most complete integration yet of search, simulation, scoring, and symbolic tuning bringing together all of our previous work on sharpening, pipeline reflection, and symbolic rules into a unified, intelligent reasoning loop.

Dimensions of Thought: A Smarter Way to Evaluate AI

Mon, 09 Jun 2025 10:00:03 +0100

📖 Summary

This post introduces a multidimensional reward modeling pipeline built on top of the stephanieanie framework. It covers:

✅ Structured Evaluation Setup How to define custom evaluation dimensions using YAML or database-backed rubrics.
🧠 Automated Scoring with LLMs Using the ScoreEvaluator to produce structured, rationale-backed scores for each dimension.
🧮 Embedding-Based Hypothesis Indexing Efficiently embedding hypotheses and comparing them for contrastive learning using similarity.
🔄 Contrast Pair Generation Creating training pairs where one hypothesis outperforms another on a given dimension.

Programming Intelligence: Using Symbolic Rules to Steer and Evolve AI

Wed, 04 Jun 2025 20:57:20 +0100

🧪 Summary

“What if AI systems could learn how to improve themselves not just at the level of weights or prompts, but at the level of strategy itself? In this post, we show how to build such a system, powered by symbolic rules and reflection.

The paper Symbolic Agents: Symbolic Learning Enables Self-Evolving Agents introduces a framework where symbolic rules guide, evaluate, and evolve agent behavior.

Adaptive Reasoning with ARM: Teaching AI the Right Way to Think

Wed, 28 May 2025 22:22:46 +0100

Summary

Chain-of-thought is powerful, but which chain? Short explanations work for easy tasks, long reflections help on hard ones, and code sometimes beats them both. What if your model could adaptively pick the best strategy, per task, and improve as it learns?

The Adaptive Reasoning Model (ARM) is a framework for teaching language models how to choose the right reasoning format direct answers, chain-of-thoughts, or code depending on the task. It works by evaluating responses, scoring them based on rarity, conciseness, and difficulty alignment, and then updating model behavior over time.