Francesco Gabellini

Is Jev calibrated?

Calvin Tran — Mon, 21 Sep 2026 00:00:00 +0000

The same experiment as confidence.ipynb (OpenAI gpt-4o-mini + logprobs), redone with TypeSafe’s Jev: 20 Newsgroups posts, classified as mac / motor / baseball, then checked with reliability curves and ECE.

What is the same: the data (test split, random_state=42, first 500 posts, truncated to 500 characters), the F1 score, the per-class reliability curves, and ECE.

What is different:

Jev’s Choice answer returns a probability for every option, so there is no logprob-token heuristic to map tokens back to classes.
The API key is read from TYPESAFE_API_KEY (or prompted for), never hardcoded.
The ECE function is fixed: the original silently dropped every prediction with probability exactly 1.0 (the most overconfident ones).
Recalibration uses Platt scaling, scored out-of-fold, and the last section compares Jev (raw and recalibrated) with the OpenAI run.

All analysis uses Jev’s probabilities. Its confidence field is a rescaled peak of that distribution, not a probability of being correct, so it is saved but not used for calibration.

import os, sys, subprocess, getpass

try:
 import typesafe_sdk
except ModuleNotFoundError:
 subprocess.check_call([sys.executable, "-m", "pip", "install", "-q", "typesafe-sdk"])
 # pip may upgrade packages this kernel already imported (e.g. typing_extensions), so a restart is required
 raise RuntimeError("typesafe-sdk was just installed. Restart the kernel (Kernel > Restart) and run all cells again.")

if not os.environ.get("TYPESAFE_API_KEY"):
 os.environ["TYPESAFE_API_KEY"] = getpass.getpass("TypeSafe API key (create one at https://console.typesafe.ai/): ")

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from concurrent.futures import ThreadPoolExecutor, as_completed
from sklearn import metrics
from sklearn.calibration import calibration_curve
from sklearn.datasets import fetch_20newsgroups
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold
from typesafe_sdk import Choice, TypeSafeClient, TypeSafeError

1. Data

MODEL = "jev-latest"
SAMPLE_SIZE = 500 # same as the OpenAI notebook
MAX_CHARS = 500 # same truncation as the OpenAI notebook
MAX_WORKERS = 8 # parallel requests; the SDK retries 429s with backoff
RESULTS_CSV = "jev_classification_results.csv"
FORCE_RECOMPUTE = False # True = ignore the cached CSV and call the API again

categories_to_fetch = ['comp.sys.mac.hardware', 'rec.motorcycles', 'rec.sport.baseball']
simplified_names = ['mac', 'motor', 'baseball']
prob_cols = [f"prob_{n}" for n in simplified_names]

fetch_kwargs = dict(remove=('headers', 'footers', 'quotes'), categories=categories_to_fetch, shuffle=True, random_state=42)
newsgroups_train = fetch_20newsgroups(subset='train', **fetch_kwargs)
newsgroups_test = fetch_20newsgroups(subset='test', **fetch_kwargs)
assert list(newsgroups_test.target_names) == categories_to_fetch # so target id i <-> simplified_names[i]

sample_data = newsgroups_test.data[:SAMPLE_SIZE]
y_true_sample = newsgroups_test.target[:SAMPLE_SIZE]
print(f"{len(sample_data)} test articles; class counts: {np.bincount(y_true_sample).tolist()}")

500 test articles; class counts: [163, 176, 161]

2. One Jev call

One Choice question. The option names are what the answer is keyed by, and their descriptions are sent to the model, so each one says what the newsgroup covers.

QUESTIONS = {
 "category": Choice(
 instructions="Which newsgroup was this post written in?",
 criteria={
 "mac": "Apple Macintosh computer hardware",
 "motor": "Motorcycles",
 "baseball": "Baseball",
 },
 )
}

def classify(client, article):
 """One Jev call -> (probability vector in simplified_names order, Jev confidence, model version)."""
 response = client.system_one(article[:MAX_CHARS], QUESTIONS, model=MODEL)
 answer = response.choices["category"]
 probs = np.array([answer.probabilities.get(n, 0.0) for n in simplified_names])
 return probs, answer.confidence, response.model

sample_article = newsgroups_train.data[20]
true_category = simplified_names[newsgroups_train.target[20]]

with TypeSafeClient() as client:
 response = client.system_one(sample_article[:MAX_CHARS], QUESTIONS, model=MODEL)
answer = response.choices["category"]

print(f"True category: {true_category}")
print(f"Jev choice: {answer.choice} (confidence {answer.confidence:.3f}, model {response.model})")
display(pd.Series(answer.probabilities, name="probability").sort_values(ascending=False).to_frame())

3. Classify the sample

Results are written to jev_classification_results.csv as soon as the calls finish, and reused on later runs (set FORCE_RECOMPUTE = True to redo them). Failed calls are counted and excluded, like the invalid predictions in the OpenAI notebook.

def run_one(client, i, article):
 try:
 probs, confidence, model = classify(client, article)
 return i, probs, confidence, model, None
 except TypeSafeError as e:
 return i, None, None, None, repr(e)

if os.path.exists(RESULTS_CSV) and not FORCE_RECOMPUTE:
 print(f"Using cached results from {RESULTS_CSV}")
else:
 rows, errors = [], {}
 print(f"Starting Jev calls for {len(sample_data)} articles...")
 with TypeSafeClient() as client, ThreadPoolExecutor(MAX_WORKERS) as pool:
 futures = [pool.submit(run_one, client, i, a) for i, a in enumerate(sample_data)]
 for done, future in enumerate(as_completed(futures), 1):
 i, probs, confidence, model, error = future.result()
 if error:
 errors[i] = error
 else:
 rows.append({"article_idx": i, "y_true_final": int(y_true_sample[i]),
 **dict(zip(prob_cols, probs)), "confidence": confidence, "model": model})
 if done % 50 == 0:
 print(f"Processed {done}/{len(sample_data)} articles.")

 if not rows:
 raise RuntimeError(f"Every Jev call failed. First error: {next(iter(errors.values()))}")
 pd.DataFrame(rows).sort_values("article_idx").to_csv(RESULTS_CSV, index=False)
 print(f"Saved {len(rows)} results to {RESULTS_CSV}; {len(errors)} calls failed.")
 if errors:
 print("First errors:", list(errors.values())[:3])

results_df = pd.read_csv(RESULTS_CSV)
print(f"Model version(s): {results_df['model'].unique().tolist()}")

Using cached results from jev_classification_results.csv
Model version(s): ['jev-1.13.0']

4. Macro F1

y_true = results_df["y_true_final"].to_numpy()
P = results_df[prob_cols].to_numpy() # (n_samples, 3) probabilities straight from Jev
y_pred = P.argmax(axis=1)

f1_jev = metrics.f1_score(y_true, y_pred, average="macro")
print("==================================================")
print(f"Macro F1 (Jev): {f1_jev:.4f} accuracy: {(y_pred == y_true).mean():.4f}")
print(f"(Calculated on {len(y_true)} valid predictions out of {SAMPLE_SIZE})")
print("==================================================")

==================================================
Macro F1 (Jev): 0.9186 accuracy: 0.9180
(Calculated on 500 valid predictions out of 500)
==================================================

5. Calibration plot (per class, one-vs-rest)

fig, ax = plt.subplots(figsize=(10, 8))
ax.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")

for i, name in enumerate(simplified_names):
 fraction_of_positives, mean_predicted_value = calibration_curve(y_true == i, P[:, i], n_bins=10)
 ax.plot(mean_predicted_value, fraction_of_positives, "o-", label=name)

ax.set_xlabel("Mean Predicted Probability (Jev)")
ax.set_ylabel("Fraction of Positives (True Probability)")
ax.set_title(f"Calibration Plot (Reliability Curve) for Jev Predictions (N={len(y_true)})")
ax.legend(loc="lower right")
ax.grid(True, linestyle="--", alpha=0.7)
plt.savefig("jev_calibration_plot.png")
plt.show()

6. Expected Calibration Error

Same definition as the original notebook, with one fix: np.digitize puts a probability of exactly 1.0 in bin index 10, which the loop never visits, so those predictions were dropped from the average. Jev (like the OpenAI mapping) returns many exact 0.0/1.0 values, and the 1.0 ones are where overconfidence shows up, so they have to be counted.

def expected_calibration_error(y_true, prob_pred, n_bins=10):
 y_true, prob_pred = np.asarray(y_true), np.asarray(prob_pred)
 bins = np.linspace(0, 1, n_bins + 1)
 binids = np.clip(np.digitize(prob_pred, bins) - 1, 0, n_bins - 1) # p == 1.0 goes in the last bin
 ece = 0.0
 for b in range(n_bins):
 mask = binids == b
 if mask.any():
 ece += abs(y_true[mask].mean() - prob_pred[mask].mean()) * mask.sum() / len(y_true)
 return ece

for i, name in enumerate(simplified_names):
 ece = expected_calibration_error((y_true == i).astype(int), P[:, i])
 print(f"ECE for '{name}': {ece * 100:.2f}%")

ECE for 'mac': 5.34%
ECE for 'motor': 4.50%
ECE for 'baseball': 1.96%

Helper used by the tables below. Top-label ECE asks whether the probability given to the predicted class matches how often that prediction is right (with a bootstrap 95% interval). Per-class ECE alone looks good when most probabilities are near 0, and ECE is biased upward at a few hundred samples, so read the interval before calling a difference real. acc p>=0.99 is the direct test of “a probability of 1.0 should be right every time”.

N_CLASSES = len(simplified_names)

def summarize(name, y, P, n_boot=1000, seed=0):
 y_pred, top_prob = P.argmax(axis=1), P.max(axis=1)
 correct = (y_pred == y).astype(int)
 rng = np.random.default_rng(seed)
 boots = [expected_calibration_error(correct[idx], top_prob[idx])
 for idx in (rng.integers(0, len(y), len(y)) for _ in range(n_boot))]
 lo, hi = np.percentile(boots, [2.5, 97.5]) * 100
 sure = top_prob >= 0.99
 row = {
 "model": name,
 "n": len(y),
 "accuracy": correct.mean(),
 "macro F1": metrics.f1_score(y, y_pred, average="macro"),
 "mean top prob": top_prob.mean(),
 "top-label ECE %": expected_calibration_error(correct, top_prob) * 100,
 "95% CI": f"[{lo:.1f}, {hi:.1f}]",
 "Brier": np.mean(np.sum((P - np.eye(P.shape[1])[y]) ** 2, axis=1)),
 "n p>=0.99": int(sure.sum()),
 "acc p>=0.99": correct[sure].mean() if sure.any() else np.nan,
 }
 for i, c in enumerate(simplified_names):
 row[f"ECE {c} %"] = expected_calibration_error((y == i).astype(int), P[:, i]) * 100
 return row

7. Platt recalibration

Platt scaling fits one sigmoid per class on the logit of Jev’s probability (2 parameters per class), then renormalises the rows to sum to 1.

It is scored out-of-fold: with 5-fold cross-validation each article is recalibrated by a model that never saw it, so all ~500 articles count and nothing is scored on the data it was fitted on. Probabilities of exactly 0 or 1 are clipped to 1e-6 first, because the logit of 0 or 1 is infinite.

EPS = 1e-6

def _logit(p):
 p = np.clip(p, EPS, 1 - EPS)
 return np.log(p / (1 - p))

def fit_platt(P_fit, y_fit):
 return [LogisticRegression(C=1e3).fit(_logit(P_fit[:, [k]]), (y_fit == k).astype(int))
 for k in range(P_fit.shape[1])]

def apply_platt(models, P_new):
 cal = np.column_stack([m.predict_proba(_logit(P_new[:, [k]]))[:, 1] for k, m in enumerate(models)])
 totals = cal.sum(axis=1, keepdims=True)
 return np.where(totals > 0, cal / np.where(totals > 0, totals, 1), 1 / cal.shape[1])

def out_of_fold_platt(P, y, n_splits=5, seed=42):
 out = np.zeros_like(P)
 for fit_idx, test_idx in StratifiedKFold(n_splits, shuffle=True, random_state=seed).split(P, y):
 out[test_idx] = apply_platt(fit_platt(P[fit_idx], y[fit_idx]), P[test_idx])
 return out

P_platt = out_of_fold_platt(P, y_true)

fig, axes = plt.subplots(1, N_CLASSES, figsize=(18, 5), sharey=True)
for i, (name, ax) in enumerate(zip(simplified_names, axes)):
 ax.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
 for probs, style, label in [(P, "o--", "Jev (raw)"), (P_platt, "s-", "Jev + Platt (out-of-fold)")]:
 frac, mean_p = calibration_curve(y_true == i, probs[:, i], n_bins=10)
 ax.plot(mean_p, frac, style, label=label, alpha=0.8)
 ax.set_title(f"'{name}' (N={len(y_true)})")
 ax.set_xlabel("Mean Predicted Probability")
 ax.grid(True, linestyle="--", alpha=0.7)
axes[0].set_ylabel("Fraction of Positives")
axes[0].legend(loc="upper left")
plt.tight_layout()
plt.savefig("jev_platt_calibration_plot.png")
plt.show()

print(f"Accuracy: raw {(P.argmax(1) == y_true).mean():.4f} | Platt {(P_platt.argmax(1) == y_true).mean():.4f}")
for i, name in enumerate(simplified_names):
 before = expected_calibration_error((y_true == i).astype(int), P[:, i]) * 100
 after = expected_calibration_error((y_true == i).astype(int), P_platt[:, i]) * 100
 print(f"ECE '{name}': {before:.2f}% -> {after:.2f}% after Platt")

Accuracy: raw 0.9180 | Platt 0.9260
ECE 'mac': 5.34% -> 2.23% after Platt
ECE 'motor': 4.50% -> 2.10% after Platt
ECE 'baseball': 1.96% -> 1.87% after Platt

8. Comparison with the OpenAI run

openai_classification_results.csv does not keep the article index (and has 496 valid rows), so the OpenAI run is a different sample of the same distribution: compare it as a reference rather than row by row. Its probabilities come from the first-token logprob heuristic in confidence.ipynb, renormalised over the matched tokens; Jev’s are its native probabilities.

runs = {
 "Jev": (y_true, P),
 "Jev + Platt (out-of-fold)": (y_true, P_platt),
}
if os.path.exists("openai_classification_results.csv"):
 openai_df = pd.read_csv("openai_classification_results.csv")
 runs["OpenAI gpt-4o-mini"] = (openai_df["y_true_final"].to_numpy(), openai_df[prob_cols].to_numpy())
else:
 print("openai_classification_results.csv not found; showing Jev only.")

summary = pd.DataFrame([summarize(name, y, probs) for name, (y, probs) in runs.items()]).set_index("model")
display(summary.round(3))

metric	Jev	Jev + Platt (out-of-fold)	OpenAI gpt-4o-mini
n	500	500	496
accuracy	0.918	0.926	0.911
macro F1	0.919	0.926	0.913
mean top prob	0.942	0.920	0.986
top-label ECE %	3.524	1.248	7.454
95% CI	[2.5, 5.8]	[1.1, 3.4]	[5.4, 9.8]
Brier	0.115	0.099	0.154
n p>=0.99	356	328	451
acc p>=0.99	0.997	0.997	0.967
ECE mac %	5.338	2.232	6.813
ECE motor %	4.498	2.102	5.224
ECE baseball %	1.962	1.868	3.437

colors = {"Jev": "tab:blue", "Jev + Platt (out-of-fold)": "tab:green", "OpenAI gpt-4o-mini": "tab:orange"}
fig, axes = plt.subplots(1, N_CLASSES + 1, figsize=(22, 5), sharey=True)

for ax, title in zip(axes, [*simplified_names, "top label (all classes)"]):
 ax.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
 ax.set_title(title)
 ax.set_xlabel("Mean Predicted Probability")
 ax.grid(True, linestyle="--", alpha=0.7)

for name, (y, probs) in runs.items():
 for i in range(N_CLASSES):
 frac, mean_p = calibration_curve(y == i, probs[:, i], n_bins=10)
 axes[i].plot(mean_p, frac, "o-", color=colors[name], label=name)
 frac, mean_p = calibration_curve(probs.argmax(1) == y, probs.max(1), n_bins=10)
 axes[-1].plot(mean_p, frac, "o-", color=colors[name], label=name)

axes[0].set_ylabel("Fraction of Positives")
axes[0].legend(loc="upper left")
plt.tight_layout()
plt.savefig("jev_vs_openai_calibration.png")
plt.show()

Classification is back

Calvin Tran — Thu, 17 Sep 2026 00:00:00 +0000

The decision was always the product

A while ago I wrote about how badly calibrated LLMs are. Any GPT-model beat a traditional ML model (TF-IDF & Naive Bayes) at classifcation but it’s probability scores are just too uncalibrated to be useful.

Then I wrote a code along on agentic patterns, and every one of those patterns has the same problem underneath. Routing is a classifier, the evaluator is a classifier, the tool selector is a classifier and so on..

Put the two articles together and you have the real problem with modern automation. What we build is workflows. What a workflow needs at every node is a decision with an honest probability attached. What the model is optimised to produce is text not calibrated decisions.

What automation actually asks for

Strip the marketing off any enterprise use case and the loop is the same. Ingest some context, most of it unstructured. Make a decision. Act on it with a tool. Repeat.

None of those steps needs the capability to produce text. Nobody in production reads the model’s reasoning trace of why it routed the ticket to the credit card team. The explanation exists because text is the only interface on offer, so we generate it and throw it away after parsing the output to generate a structured response.

The parsing is the tell. If your first action on receiving a model output is to run a validator over it, the model is not speaking your language.

The hoops

Everyone building these systems carries the same scaffolding. A Pydantic schema plus some additional tricks , because the model can emit a category that does not exist. Temperature zero, which does not make the system deterministic, it just makes it feel deterministic. Retry loops, so you make five calls instead of one.

This is not a prompting problem. Guo et al. (2017) showed that modern networks drift towards overconfidence as a property of how we train them, and RLHF pushes the same way by optimising for what a human rater prefers to read. A model rewarded for sounding helpful will sound certain.

System One models

TypeSafe released Jev on 15 September, the first of what they call System One models. The name comes from Kahneman’s split between fast intuitive judgement and slow deliberate reasoning.

The design decision is the interesting part. Jev gives up text generation entirely. It samples all of its outputs in parallel in a single query rather than one token at a time, and it is trained with a method they call Reinforcement Learning for Calibrated Decisions instead of RLHF. You send a state and a set of typed questions, and you get back typed values and probability distributions.

Everything in the previous section becomes unnecessary if that holds. There is no parsing because there is no string. There are no type errors, not because they are rare but because the possible outputs are defined in advance and schema matching is guaranteed. And the probability is not fugazi, it is what the model was trained to produce.

Calibration is king

Calibration means that among all the predictions the model assigns 70% to, roughly 70% turn out correct. Without that property you cannot set a threshold, and without a threshold every decision needs a human in the loop.

Jav returns the full distribution and also collapses its shape into a single confidence number, so you can threshold without computing it yourself. A flat distribution means low confidence: no option is a clear winner, or the state does not contain enough to go on. What follows is the pattern every risk function already understands. A floor below which you route to a human, and above it a bar for acting without confirmation that scales with what the action costs if it is wrong.

The part I am not sold on yet

The workflow evals are the weak point, agreement with a panel of frontier LLMs is not the same as being right (try LLM as a judge), and it is a strange yardstick for a model whose pitch is that frontier LLMs are badly calibrated. The one claim genuinely free of this problem is type safety, which is guaranteed by construction and falsifiable with a single API call.

And the obvious one. This does not replace LLMs. It replaces the part of your agent that was pretending to be a classifier. When you need generated language,images or video you still need a generative model. What changes is that the decision layer stops being made of the same material as the generation layer.

What I want to see, and intend to run myself (if I get my hand on the early access), is a straight calibration benchmark. Same 20 Newsgroups setup as the first article, reliability diagram, ECE, Brier score, no vendor in the loop.

Conclusion

I ended the calibration article by saying that where a decision depends on a trustworthy measure of confidence, modern LLMs are a bad choice, and you should accept the cost of labels and training something simpler. I still think that was right for the world as it was.

But if the generality (of inputs) and the calibration (of outputs) hold up across common domains, RLCD could turn out to be the missing piece in every automation workflow.

And I would be very happy to delete the scaffolding we built around LLMs to force them into being type safe and marginally less overconfident, and to pay less for the privilege.

Who Watches the Watchmen?

Calvin Tran — Sun, 21 Jun 2026 00:00:00 +0000

This article is about single-turn LLM generation: producing text, answering questions, summarising, and so on. I am not going to cover multi-turn conversation or agentic systems. The question is a practical one. You have shipped, or are about to ship, a system that relies on an LLM for real users in a real business. How do you know if it is any good?

The old world: NLP measures

Before large language models entered the picture, NLP evaluation had a clean, mathematical answer: measure how similar the output is to a known good reference. The dominant metric was BLEU (Papineni et al., 2002), the Bilingual Evaluation Understudy, imported from the machine translation research that would later give us the Transformer (Vaswani et al., 2017). BLEU counts overlapping n-grams between a generated string and a reference string and produces a score between 0 and 1. ROUGE (Lin, 2004) did the same for summarisation. Cosine similarity over TF-IDF or word embeddings offered a softer version of the same idea.

These metrics had real virtues. They were fast, cheap, deterministic, and needed no human in the loop once you had a reference corpus.

The problem is that language is not a bag of words, and meaning is not proximity in token space. “The patient is not responding to treatment” and “The patient is responding well to treatment” share most of their surface and would score highly against one another. “The medication was administered” and “The medication was not administered” sit a single negation apart. Similarity-based metrics are structurally blind to these distinctions, and in a real world scenario that blindness becomes fatal.

Closing the gap: Natural Language Inference

The field’s next attempt to do better came through Natural Language Inference (NLI): the task of deciding whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise. Models fine-tuned on datasets like SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018) learned something closer to semantic reasoning than to surface matching.

The training data looks like this:

Premise	Hypothesis	Label
A football game with kids playing.	Some kids are playing.	Entailment
A man is in a kitchen cooking	The man is sleeping.	Contradiction

The idea, applied to evaluation, was to check not whether the output looked like the reference, but whether it was logically coherent with, it was a genuine improvement. NLI-based scoring caught contradictions that BLEU missed; it was sensitive to negation, to entailment direction, to whether a claim was supported or merely adjacent to the reference.

But NLI brought its own problem, and you can already see it in the table above. Those premises and hypotheses are short, tidy, single-clause sentences derived from image captions. Real documents do not read like the Elements of Euclid. Real world documents are messy, contextual, sentences long, full of hedges and asides and clauses that depend on one another. An entailment model trained on caption pairs has very little to say about whether a four-paragraph answer faithfully reflects a retrieved policy document. It was a well-designed experiment that, for the most part, stayed in the lab.

The new world : LLM-as-Judge

Once capable LLMs existed, the obvious move was to use them to evaluate each other. Feed the system prompt, the user query and the model’s response to a capable judge model and get back a score. This approach, LLM-as-Judge, was studied systematically by Zheng et al. (2023) and has since become the de facto standard. A capable judge can assess fluency, faithfulness, relevance and tone in a single pass, and it can explain its reasoning on edge cases that would confuse any fixed metric.

A growing ecosystem of packages wraps this pattern into something you can drop into a pipeline. RAGAS (Es et al., 2024) is one example among many: built for retrieval-augmented generation, it scores dimensions like faithfulness (does the answer stick to the retrieved context?), answer relevancy and context recall, each judged independently so you get a full picture rather than one opaque number. DeepEval, TruLens and others occupy the same space with different trade-offs. The framework matters less than understanding what it is doing underneath, which is asking an LLM to grade another LLM.

That is where the real problem lives, and it is one of epistemology: you are using a model with its own biases to evaluate a model with its own biases. Three failure modes are well documented. Self-preference bias: judges favour outputs that resemble their own, and Panickssery et al. (2024) link this directly to a model’s ability to recognise its own generations, so using GPT-4 to judge GPT-4 is not a neutral act. Verbosity bias: longer, more hedged answers score higher than concise, correct ones.Position bias: in a pairwise comparison the answer shown first wins more often.

Mitigations exist: prompt the judge to reason before scoring, run each comparison in both orders and average, use a different model family as judge than the one under test. None of them fully closes the gap. LLM-as-Judge is powerful tool but has it’s cracks.

Back to basics: the golden dataset

The canonical machine learning answer is the test dataset: a golden dataset of inputs paired with verified outputs, against which any version of the system can be scored and compared. The appeal is obvious. It is deterministic, reproducible and immune to judge bias.

The problem is circular in an uncomfortable way. To score an LLM’s free-form generation against a golden answer, you need a method to compare the produced response to the expected one, and you are right back where you started. A similarity metric misses semantic divergence; an NLI model carries the problems described above; a human reviewing every pair does not scale. You can use an LLM judge for the comparison, but then your golden evaluation is only as good as that judge, and you have not escaped the watchmen problem, you have just moved it up one layer.

Golden datasets still earn their place when you narrow what they test. For constrained tasks (extracting a specific field, classifying into a fixed set of categories, requiring a response to contain certain factual claims) binary or near-binary correctness checks are both tractable and meaningful. The failure mode is stretching this to open-ended generation, where the space of acceptable answers is large and your reference is only one valid output among many.

When you do need to measure closeness to an expected answer, the most reliable options today are embedding-based similarity using a strong general-purpose embedding model, or structured decomposition: break the expected answer into individual factual claims and verify each one separately rather than comparing the whole.

The user as ground truth

There is one more signal, and it sits in production: the user. Thumbs up and thumbs down, copy-to-clipboard events, a follow-up question that reveals the first answer was wrong, session usage. These are all expressions of satisfaction or dissatisfaction that need no instrumentation beyond basic analytics.

The seductive thing about implicit feedback is that it is free, continuous and available at scale. The dangerous thing is that it is confounded by almost everything: UI, task difficulty, user expertise, stylistic preferences that have nothing to do with quality. A thumbs down might mean “the answer was wrong”, or “the answer was right but too boring”, or “I clicked it by accident”, or “I was already frustrated before I opened the tool”. Explicit Likert ratings remove some of that ambiguity but add survey fatigue and selection bias, since the users who rate are rarely representative of the users who do not.

Treat user feedback as a monitoring signal, not a validation metric. It is good for catching regressions, surfacing categories of failure and deciding what to investigate next. It is not reliable enough to be the primary basis for an iteration decision.

A practical framework

Putting it all together, here is a reasonable approach for a team shipping a production LLM system, at least in my opinion

Define what correctness means before you build anything. The evaluation strategy should fall out of the task definition, not get bolted on afterwards. If you cannot write down in concrete terms what a good response looks like, you are not ready.

Use automated metrics as filters, not as truth. LLM-as-Judge metrics catch obvious failures reliably and subtle ones unreliably. Use them to shrink the volume of output that needs human eyes, not to replace those eyes.

Build a golden dataset for regression testing A modest, carefully curated set of inputs with verified references beats a large noisy one.That set is your foundation for changing the system without breaking it in production.

Show the retrieved context or the reasoning trace to the user. If the job is to convey accurate information, expose the sources behind each answer so the user can confirm or reject them. This turns your users into a distributed check on your failure modes.

Treat user feedback as continuous monitoring. Log it, track it over time, investigate anomalies, and keep iterating.

Evaluate offline first, then ship with monitoring. The ship of certainty sailed long ago. The goal is not to be perfect before deployment, which is unattainable, but to have enough signal to catch a mistake quickly and roll back.

Stay sceptical of every metric All of this is probabilistic. No single number tells you the system is good; you are watching whether the whole picture is trending in the right direction. Any metric can be gamed, can drift, or can look healthy while something underneath rots. Trust the convergence of several weak signals over any one strong-looking score.

Conclusion

There is no clean solution to this. Every evaluation method has a crack, and every crack is filled by some other method you are trusting implicitly. What you can do is be honest about where each layer breaks down, stack layers that fail in different ways, and keep humans in contact with real output often enough to catch what the metrics miss. That is not a perfect answer. It is the only one on offer.

References

Papineni, K., Roukos, S., Ward, T., Zhu, W. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation. ACL.
Lin, C. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out, ACL Workshop.
Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS.
Bowman, S., Angeli, G., Potts, C., Manning, C. (2015). A Large Annotated Corpus for Learning Natural Language Inference. EMNLP.
Williams, A., Nangia, N., Bowman, S. (2018). A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. NAACL.
Zhang, T., Kishore, V., Wu, F., Weinberger, K., Artzi, Y. (2020). BERTScore: Evaluating Text Generation with BERT. ICLR.
Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS.
Es, S., James, J., Espinosa-Anke, L., Schockaert, S. (2024). RAGAS: Automated Evaluation of Retrieval Augmented Generation. EACL.
Panickssery, A., Bowman, S., Feng, S. (2024). LLM Evaluators Recognize and Favor Their Own Generations. NeurIPS.

PageIndex: When Your RAG Reads Like a Human

Calvin Tran — Tue, 12 May 2026 00:00:00 +0000

The Explainability Problem Nobody Talks About

Standard RAG has a transparency problem, and it’s not one that shows up in benchmark numbers.

When a user asks “why did the system return this paragraph and not that one?”, the honest answer is: “because the cosine similarity was 0.83 instead of 0.79.” That answer is technically correct and completely useless. Nobody outside of an ML team reasons about document retrieval in terms of vector distances. They reason about sections, chapters, topics, and arguments.

This gap between how retrieval systems work and how humans think about information lookup creates a problem: the users either accept the result blindly or reject the whole system because they can’t audit it. Neither is good.

PageIndex is an approach that sidesteps this problem entirely by discarding vector similarity and replacing it with something more explainable : structure-aware, reasoning-based retrieval.

How Humans Actually Look Things Up

If you remember searching for something in a physical library before Google, I do, you remember the process intuitively. You didn’t scan every page of every book. You:

Found the right book by subject
Opened the table of contents
Found the relevant chapter
Skimmed the section headings
Read the specific paragraph

That multi-level navigation was precise and fully explainable. You could tell anyone exactly why you landed on page 247 of Option futures and other derivatives.

PageIndex replicates this exact behavior. Instead of embedding chunks and retrieving by similarity, it first builds a hierarchical index of the documents. Baiscally a machine-readable table of contents with summaries at each node and then asks an LLM to reason about which branch of that tree is likely to contain the answer. The LLM walks the index the same way I used to serarch the uni library.

Two Phases: Index Then Retrieve

The workflow has two clean phases.

Phase 1 — Build the index. Submit a document to PageIndex. It parses the document structure (headings, sections, subsections), generates a summary for each node, and returns a JSON tree. This happens once per document.

Phase 2 — Reason and retrieve. At query time, pass the tree (without full text) to an LLM and ask it to identify which nodes are relevant. Then pull the actual text only from those nodes and generate the final answer.

The tree for a typical long document looks like this:

Document Root
├── Abstract [pages 1-1]
│ └── "Overview of the proposed method and key results..."
├── Introduction [pages 2-3]
│ └── "Motivation, problem statement, and prior work..."
├── Methodology [pages 4-8]
│ ├── Data Collection [pages 4-5]
│ ├── Model Architecture [pages 5-7]
│ └── Training Procedure [pages 7-8]
├── Results [pages 9-12]
│ ├── Quantitative Evaluation [pages 9-11]
└── Conclusion [pages 11-12]
└── "Summary of findings and future directions..."

The LLM sees this structure with the summaries and can reason: “the question is about conclusions, so I should look at the Conclusion node.” That reasoning is visible, auditable, and directly explainable to any user.

The Code

Setup is minimal : Clone, install dependencies, and set your LLM key. Then generate the tree locally from a PDF :

python3 run_pageindex.py --pdf_path document.pdf \
 --if-add-node-summary yes \
 --if-add-node-text yes

Load the result and set up your LLM client:

import pageindex.utils as utils
import openai, json

with open("results/document_structure.json") as f:
 tree = json.load(f)

utils.print_tree(tree)

def call_llm(prompt, model="gpt-4.1", temperature=0):
 client = openai.AsyncOpenAI()
 response = await client.chat.completions.create(
 model=model,
 messages=[{"role": "user", "content": prompt}],
 temperature=temperature
 )
 return response.choices[0].message.content.strip()

At query time, strip the full text out of the tree, show it to the LLM, and ask for the relevant nodes:

query = "What are the main conclusions?"

tree_without_text = utils.remove_fields(tree.copy(), fields=['text'])

search_prompt = f"""
You are given a question and a tree structure of a document.
Each node contains a node id, title, and summary.
Find all nodes likely to contain the answer.

Question: {query}
Document tree: {json.dumps(tree_without_text, indent=2)}

Reply in JSON:
{{
 "thinking": "<your reasoning>",
 "node_list": ["node_id_1", "node_id_2"]
}}
"""

tree_search_result = call_llm(search_prompt)

Then extract text only from the selected nodes and generate the answer:

node_map = utils.create_node_mapping(tree)
result = json.loads(tree_search_result)

node_list = result["node_list"]
relevant_content = "\n\n".join(node_map[nid]["text"] for nid in node_list)

answer = await call_llm(f"Answer based on context:\n\nQuestion: {query}\nContext: {relevant_content}")

Stateless by Design, Storable if Needed

Because the index is just JSON, the whole system is flexible about state. You can run it entirely in memory for small queries, or persist the tree to disk, a database, or an object store, you do you.

import json

# persist
with open("doc_tree.json", "w") as f:
 json.dump(tree, f)

# restore
with open("doc_tree.json") as f:
 tree = json.load(f)

This is meaningful in practice. Vector databases introduce a whole infrastructure layer: embedding models, database extensions, fancy fusion algorithm. PageIndex’s index is a JSON file. It integrates with whatever you already have,an S3 bucket, a Redis cache,just some memory. No additional infrastructure required.

Structure-Aware Chunking vs Fixed-Size Chunking

This is perhaps the most underappreciated difference.

Traditional RAG cuts documents into fixed-size chunks, typically 512 or 1024 tokens, with some overlap to avoid losing context. The problem is that documents are not uniformly structured. A single section might be 200 tokens; another might be 3000. A fixed-size chunk will routinely split a coherent argument mid-sentence or merge the end of one section with the start of an unrelated one.

Traditional RAG chunking (fixed-size):
│◄── 512 tokens ──►│◄── 512 tokens ──►│◄── 512 tokens ──►│
[intro...][ch1 end][ch2 start...mid ][...ch2 end][ch3...]
PageIndex chunking (structure-aware):
│◄── Introduction ──►│◄─────── Chapter 2 ─────────►│◄── Chapter 3 ──►│
[coherent unit ] [coherent unit ] [coherent unit ]

Structure-aware chunks are semantically complete. When shown to the user as a source reference, they read like a passage — not like a sentence that got cut off because a token counter hit a limit.

Scaling Limitations

PageIndex is not a silver bullet. It is fundamentally an in-context approach: the entire tree (without text) is passed to the LLM at query time. For very long documents with deep, granular structure, this tree can itself become large. The LLM must process it in a single context window.

It is worth being honest about this. But it is equally worth being honest about the fact that standard RAG also does not excel at scale. Top-k retrieval over large corpora produces noisy results; relevance degrades as the corpus grows and context bloating is real.

For the use cases where it fits: long structured documents, regulatory filings, technical manuals, research papers, contracts PageIndex is genuinely strong. FinanceBench results report ~98.7% accuracy on financial document QA.

Conclusion

PageIndex is a good example of a retrieval approach that optimises for the right thing. Not just accuracy, but explainability. The retrieval path is a reasoning trace. The chunks are coherent sections. The index is a plain JSON file.

None of this is magic. It is essentially the same thing a well-organised student does when they open a textbook. The insight is just recognising that human information retrieval is already an excellent algorithm, and that replicating it is could be better than replacing it with cosine similarity.

The official repo and cookbook are at github.com/VectifyAI/PageIndex if you want to run the full notebook.

As a recovering statistician, I am profoundly happy when I can make an AI system a bit more explainable, and profoundly unhappy when I can’t.

Do Foundational Models Actually Work for Forecasting?

Calvin Tran — Sat, 18 Apr 2026 00:00:00 +0000

Traditional forecasting

Every forecasting practitioner from a math/stats background knows the ritual. You get a new series, run the ADF test, differentiate, and hope that the series is stationary. You plot the ACF and PACF, count the significant lags, make a guess at p and q, run a grid search over AIC, wait, then, once you have figured out the period, you do it all over again with a seasonal component.

For the uninitiated, you can find a more in-depth explanation in the bible of forecasting: Forecasting: Principles and Practice

# Step 1: determine d via ADF + KPSS
def find_d(series, max_d=2):
 ser = series.copy()
 for d in range(max_d + 1):
 adf_p = adfuller(ser.dropna(), autolag="AIC")[1]
 kpss_p = kpss(ser.dropna(), regression="c", nlags="auto")[1]
 if (adf_p < 0.05) and (kpss_p > 0.05):
 return d
 ser = ser.diff()
 return max_d

# Step 2: detect seasonal period from the periodogram
s = detect_season(m4_series)

# Step 3: AIC grid search over (p,q) x (P,Q)
for (p, q), (P, Q) in itertools.product(search_pq, search_PQ):
 aic = SARIMAX(data, order=(p, d, q),
 seasonal_order=(P, D, Q, s)).fit(disp=False).aic
 ...

If you wanted to modernize your forecasting approach and follow what practitioners on Kaggle are doing, you could switch to a machine learning model like LightGBM. However, with LightGBM, you first need to engineer lag features, decide how many lags to include, add rolling statistics, and tune hyperparameters such as num_leaves, learning_rate, and subsample using tools like Optuna before you can even think of generating a single forecast.

study = optuna.create_study(direction="minimize",
 sampler=optuna.samplers.TPESampler(seed=42))
study.optimize(objective, n_trials=40, show_progress_bar=True)
best_lgb_params = study.best_params

The problem is worse than just tedium: none of this work transfers between projects. You accumulate better instincts and reusable code snippets, but every new series forces you to restart the entire process from scratch.

Deep learning did not solve the problem

The obvious hope was that deep learning would do what it did for images and text: learn a universal representation and make the hand-crafted pipeline obsolete. It did not work out that way.

The M4 competition (2018), covering 100,000 real-world series, was the first large-scale empirical test. The winner was a hybrid ES-RNN, not a pure deep learning model, but a classical Exponential Smoothing model with an RNN on top. Pure neural methods consistently underperformed compared to statistical baselines that had been around for decades.

The M5 and M6 competitions confirmed the pattern. The use of LightGBM with careful feature engineering repeatedly beat deep learning architectures. The community largely concluded that deep learning was simply too cumbersome to train on short series, too prone to overfitting, and too sensitive to architecture choices to be worth the cost of training.

The fundamental problem was that every deep learning model still needed to be trained from scratch on each dataset. It was essentially a more expensive version of the same ritual.

What TimesFM changes

Google Research published A decoder-only foundation model for time-series forecasting in 2023. I’ve noticed the advances in transformer models when they started to creep into the industry I work in, aka banking, but only recently, while reading the paper PRAGMA: Revolut Foundation Model, did it click. The core idea is simple: pre-train a large transformer on an enormous and diverse corpus of time series data, then use it zero-shot on any new series — no hyperparameter search, no statistical tests.

TimesFM is to forecasting what GPT was to NLP: a single pre-trained model that generalises across tasks it has never seen.

This is the part that matters for practitioners. After all the machinery described above — stationarity tests, seasonal detection, AIC search, Optuna — the TimesFM inference path is:

import timesfm

tfm = timesfm.TimesFm(
 hparams=timesfm.TimesFmHparams(
 backend="torch",
 horizon_len=14,
 context_len=512,
 ),
 checkpoint=timesfm.TimesFmCheckpoint(
 huggingface_repo_id="google/timesfm-1.0-200m-pytorch"
 ),
)

point_forecast, _ = tfm.forecast(inputs=[train_data], freq=[0])

There is no fit(). The model has never seen your series. You hand it the historical values, and it returns a forecast. The entire statistical/ML ritual is replaced by a forward pass through a pre-trained network.

Running the same backtesting as SARIMA and LightGBM

To make this a fair comparison, all three models were evaluated on M4 Daily series D2047 — 8,533 daily observations — using identical TimeSeriesSplit cross-validation with five folds and a 14-step forecast horizon.

from sklearn.model_selection import TimeSeriesSplit

n_splits = 5
test_size = 14
tscv = TimeSeriesSplit(n_splits=n_splits, test_size=test_size)

# SARIMA: fit a new model per fold with the tuned (p,d,q)(P,D,Q,s)
for fold, (train_idx, test_idx) in enumerate(tscv.split(m4_series)):
 fit = SARIMAX(train_data, order=best_order,
 seasonal_order=best_sorder).fit(disp=False)
 preds = fit.forecast(steps=test_size).values

# LightGBM: train a new model per fold with Optuna-tuned params
for fold, (train_idx, test_idx) in enumerate(tscv.split(m4_series)):
 model = lgb.LGBMRegressor(**best_lgb_params)
 model.fit(X_tr, y_tr)
 # recursive prediction ...

# TimesFM: no training, same context each fold
for fold, (train_idx, test_idx) in enumerate(tscv.split(m4_series)):
 point_forecast, _ = tfm.forecast(inputs=[train_data], freq=[0])
 preds = point_forecast[0]

SARIMA requires running the full stationarity and order-selection pipeline before the loop. LightGBM requires several Optuna trials on an inner cross-validation. TimesFM requires none of that — the same three-line setup serves every fold unchanged.

The result: TimesFM achieves competitive MAE without a single line of training code, while SARIMA and LightGBM each needed a bespoke calibration pipeline just to participate.

Why this could democratise forecasting

TimesFM collapses the forecasting stack. A developer with no background in time-series statistics can load the model, pass in a dataframe, and get a probabilistic forecast within seconds.

This mirrors exactly what happened to NLP. Before the transformer era, building a production sentiment classifier for customer service calls meant curating internal transcriptions, collecting customer feedback labels, engineering linguistic features, selecting and tuning a model, and repeating this process for every new business domain or language. After GPT, it is possible to load a pre-trained model, pass in your call transcripts, and get sentiment predictions without annotation pipelines. The barrier to entry collapsed by an order of magnitude, and the volume of NLP applications in production exploded.

The limits are real

I didn’t spend five years studying statistical theory and another five as a data scientist just to watch it all get dismissed. Time series is weird in ways that GPT’s text never had to deal with. Financial returns don’t behave like energy demand. Sensor data from a factory floor look nothing like web traffic. The distribution shift between Google’s pre-training corpus and your specific problem can be absolutely massive.

Fine-tuning on domain-specific data can help close much of that gap, and Google’s own benchmarks show that even a few hundred in-domain examples significantly improve TimesFM’s accuracy. But that is still a fundamentally different burden than training SARIMA from scratch — it is adaptation, not construction.

Conclusion

Over the past decade, the pattern in machine learning has been consistent: provided that a large enough pre-training corpus is available, foundation models eventually outperform or at the very least match task-specific models trained from scratch for a fraction of the deployment cost. Text, images, audio, code — the story is always the same.

Time series proved more resistant simply because the data is messier, the frequency and domain vary wildly, and the benchmarks were designed around the assumption that every model would be trained from scratch. TimesFM, and the family of models following it, challenge this core assumption.

Whether TimesFM specifically becomes the standard, or is surpassed by the next generation of temporal foundation models, the direction is now clear. The era of bespoke per-series models is ending.

Building effective agents

Calvin Tran — Wed, 10 Dec 2025 00:00:00 +0000

Agentic pattern

We’re doing something different today. Instead of a typical article, this is a code along tutorial based on Anthropic’s foundational blogpost on agentic patterns : Building effective agents.

The objective of this article is to demystify the complexity of agentic workflows by presenting a simple and clear Python implementation of the core concepts defined in the Anthropic article.

0) Set-up

Replicating this exercise is possible with any LLM provider that is compatible with the Openai Client library and the set-up is minimal. We just need the LLM client, pydantic for structured output and json to parse the response.

# --- Libraries ---
import openai
from pydantic import BaseModel
from pydantic import BaseModel, Field
import json
from typing import Literal, TypedDict

1) Prompt chaining

Prompt chaining decomposes a task into a sequence of steps, where each LLM call processes the output of the previous one.

In this simple example we are going to create a chain (sequence) of prompts (generate_joke,improve_joke,polish_joke) in order to generate an output (a joke).

The chain represents the simplest workflow pattern, marking the first step in complexity beyond standard one-shot prompting for generating answers.

# --- State ---
state = {
 "topic": "",
 "joke": "",
 "improved_joke": "",
 "final_joke": ""
}
# --- Functions ---
def generate_joke(topic: str) -> str:
 response = client.chat.completions.create(
 messages=[{"role": "user", "content": f"Write a joke about {topic} with just single a sentence"}],
 )
 return response.choices[0].message.content

def improve_joke(joke: str) -> str:
 response = client.chat.completions.create(
 messages=[{"role": "user", "content": f"Make this joke funnier by adding wordplay or puns: {joke}"}],
 )
 return response.choices[0].message.content

def polish_joke(joke: str) -> str:
 response = client.chat.completions.create(
 messages=[{"role": "user", "content": f"""
 Write a short, funny joke in the following format:
 Why did [subject] [action]?
 To [funny reason related to the subject]!
 The joke should be based on the following : {joke}
 """}],
 )
 return response.choices[0].message.content

def check_punchline(joke: str) -> bool:
 return joke.count("?") >= 1

# --- Workflow Execution ---
def run_workflow(topic: str):
 state["topic"] = topic
 state["joke"] = generate_joke(topic)
 state["improved_joke"] = improve_joke(state["joke"])
 state["final_joke"] = polish_joke(state["improved_joke"])
 if check_punchline(state["final_joke"]):
 print(state["final_joke"])
 else :
 print("joke did not passed quality gate")
 return state
# --- Run ---
result = run_workflow("cats at work")

2) Routing

Routing classifies an input and directs it to a specialized followup task.

In this example, we use an initial LLM call to determine the next step whether to generate a poem, a story, or a joke, and then execute only the selected route.

The routing pattern is particularly interesting because it leverages specialization in prompts and data sources. Instead of having one generalist ‘agent’ that knows everything, it is more efficient to employ multiple specialized agents that the router can direct tasks to as needed.

# --- Schema for routing ---
class Route(BaseModel):
 step: Literal["poem", "story", "joke"] = Field(
 None, description="The next step in the routing process"
 )
# --- State ---
class State(TypedDict):
 input: str
 decision: str
 output: str
# --- Router using structured output ---
def llm_call_router(state: State) -> dict:
 response = client.chat.completions.create(
 messages=[
 {"role": "system", "content": "Route the input to story, joke, or poem based on the user's request."},
 {"role": "user", "content": state["input"]},
 ],
 response_format={
 "type": "json_schema",
 "json_schema": {
 "name": "route",
 "schema": {
 "type": "object",
 "properties": {
 "step": {
 "type": "string",
 "enum": ["poem", "story", "joke"]
 }
 },
 "required": ["step"]
 }
 }
 },
 )
 decision = json.loads(response.choices[0].message.content)
 return {"decision": decision["step"]}
# --- Nodes ---
def llm_call_story(state: State) -> dict:
 response = client.chat.completions.create(
 messages=[{"role": "user", "content": f"Write a story about: {state['input']}"}],
 )
 return {"output": response.choices[0].message.content}

def llm_call_joke(state: State) -> dict:
 response = client.chat.completions.create(
 messages=[{"role": "user", "content": f"Write a joke about: {state['input']}"}],
 )
 return {"output": response.choices[0].message.content}

def llm_call_poem(state: State) -> dict:
 response = client.chat.completions.create(
 messages=[{"role": "user", "content": f"Write a poem about: {state['input']}"}],
 )
 return {"output": response.choices[0].message.content}

# --- Routing Logic ---
def route_decision(state: State) -> str:
 if state["decision"] == "story":
 return "story"
 elif state["decision"] == "joke":
 return "joke"
 elif state["decision"] == "poem":
 return "poem"

# --- Workflow Execution ---
def run_workflow(user_input: str):
 state: State = {"input": user_input, "decision": "", "output": ""}
 # Step 1: Route
 route_result = llm_call_router(state)
 state["decision"] = route_result["decision"]
 # Step 2: Execute based on decision
 if state["decision"] == "story":
 state.update(llm_call_story(state))
 elif state["decision"] == "joke":
 state.update(llm_call_joke(state))
 elif state["decision"] == "poem":
 state.update(llm_call_poem(state))
 return state

# --- Run ---
result = run_workflow("Write me a short story about cats at work")

3) Parallelization

Breaking a task into independent subtasks run in parallel.

Unlike the previous example, which decided between a poem, a story, or a joke, this case involves generating all three in parallel and then aggregating the results.

This is a typical pattern across computer science: when a problem is too complex or slow for a single computation, it is broken down into simpler ones to achieve greater efficiency and reduced processing time.

# --- State ---
class State(TypedDict):
 topic: str
 joke: str
 story: str
 poem: str
 combined_output: str

# --- LLM Call Functions ---
def call_llm(prompt: str) -> str:
 response = client.chat.completions.create(
 messages=[{"role": "user", "content": prompt}],
 )
 return response.choices[0].message.content

def call_llm_1(state: State) -> dict:
 return {"joke": call_llm(f"Write a joke about {state['topic']}")}

def call_llm_2(state: State) -> dict:
 return {"story": call_llm(f"Write a story about {state['topic']}")}

def call_llm_3(state: State) -> dict:
 return {"poem": call_llm(f"Write a poem about {state['topic']}")}

def aggregator(state: State) -> dict:
 combined = f"Here's a story, joke, and poem about {state['topic']}!\n\n"
 combined += f"STORY:\n{state['story']}\n\n"
 combined += f"JOKE:\n{state['joke']}\n\n"
 combined += f"POEM:\n{state['poem']}"
 return {"combined_output": combined}

# --- Workflow Execution ---
def run_workflow(topic: str) -> State:
 state: State = {"topic": topic, "joke": "", "story": "", "poem": "", "combined_output": ""}
 # Run tasks in parallel (or sequentially for simplicity)
 state.update(call_llm_1(state))
 state.update(call_llm_2(state))
 state.update(call_llm_3(state))
 # Aggregate results
 state.update(aggregator(state))
 return state

# --- Run ---
result = run_workflow("cats")

4) Evaluator-optimizer

In the evaluator-optimizer workflow, one LLM call generates a response while another provides evaluation and feedback in a loop.

In this example, we implement a feedback loop: the evaluator grades the generated joke and returns feedback to the generator until the resulting joke is deemed satisfactory.

This is a highly interesting pattern as it represents a significant step toward true autonomy. In this case, two ‘agents’ collaboratively ’think’ about a problem until they reach consensus that it is solved, operating without any human intervention or feedback.

class State(TypedDict):
 joke: str
 topic: str
 feedback: str
 funny_or_not: str

# --- Schema for evaluation ---
def llm_call_evaluator(state: State) -> dict:
 response = client.chat.completions.create(
 messages=[
 {"role": "system", "content": "Grade the joke as funny or not funny and provide feedback if needed."},
 {"role": "user", "content": f"Joke: {state['joke']}"}
 ],
 response_format={
 "type": "json_schema",
 "json_schema": {
 "name": "feedback",
 "schema": {
 "type": "object",
 "properties": {
 "grade": {"type": "string", "enum": ["funny", "not funny"]},
 "feedback": {"type": "string"}
 },
 "required": ["grade", "feedback"]
 }
 }
 },
 )
 result = json.loads(response.choices[0].message.content)
 return {"funny_or_not": result["grade"], "feedback": result["feedback"]}

# --- Joke Generator ---
def llm_call_generator(state: State) -> dict:
 if state.get("feedback"):
 prompt = f"Write a joke about {state['topic']} but improve it based on this feedback: {state['feedback']}"
 else:
 prompt = f"Write a joke about {state['topic']}"
 response = client.chat.completions.create(
 messages=[{"role": "user", "content": prompt}],
 )
 return {"joke": response.choices[0].message.content}

# --- Workflow ---
def run_workflow(topic: str, max_loops: int = 5):
 state: State = {"topic": topic, "joke": "", "feedback": "", "funny_or_not": ""}
 for i in range(max_loops):
 # Generate joke
 state.update(llm_call_generator(state))
 # Evaluate joke
 eval_result = llm_call_evaluator(state)
 state.update(eval_result)
 if state["funny_or_not"] == "funny" and i>0 :
 print(" -- Joke accepted! --")
 break
 else:
 print("Joke rejected, improving...")
 return state

# --- Run ---
result = run_workflow("cats at work")

5) Agents

Agents are emerging in key capabilities—understanding complex inputs, engaging in reasoning and planning, using tools reliably, and recovering from errors.

Finally, we demonstrate an example of a truly autonomous agent. Though limited to arithmetic tools in this case, the agent can receive a task, reason, plan which tools to use, execute those tools, and autonomously stop the reasoning process when it is ready to generate the final answer to the user.

# --- Define Python functions ---
def multiply(a: int, b: int) -> int:
 return a * b

def add(a: int, b: int) -> int:
 return a + b

def divide(a: int, b: int) -> float:
 return a / b

# --- Define tool schemas ---
tools = [
 {
 "name": "multiply",
 "description": "Multiply two integers",
 "parameters": {
 "type": "object",
 "properties": {
 "a": {"type": "integer", "description": "First integer"},
 "b": {"type": "integer", "description": "Second integer"}
 },
 "required": ["a", "b"]
 }
 },
 {
 "name": "add",
 "description": "Add two integers",
 "parameters": {
 "type": "object",
 "properties": {
 "a": {"type": "integer", "description": "First integer"},
 "b": {"type": "integer", "description": "Second integer"}
 },
 "required": ["a", "b"]
 }
 },
 {
 "name": "divide",
 "description": "Divide two integers",
 "parameters": {
 "type": "object",
 "properties": {
 "a": {"type": "integer", "description": "First integer"},
 "b": {"type": "integer", "description": "Second integer"}
 },
 "required": ["a", "b"]
 }
 }
]
# --- Map tool names to Python functions ---
tools_by_name = {
 "multiply": multiply,
 "add": add,
 "divide": divide
}
import json

# --- Workflow ---
def run_agent(user_input: str, max_iterations: int = 5):
 messages = [
 {"role": "system", "content": "You are a helpful assistant tasked with performing arithmetic on a set of inputs."},
 {"role": "user", "content": user_input}
 ]
 for i in range(max_iterations):
 response = client.chat.completions.create(
 model=model,
 messages=messages,
 tools=tools,
 tool_choice="auto"
 )
 last_message = response.choices[0].message
 tool_calls = last_message.tool_calls
 # Keep the last LLM message in history
 messages.append({"role": "assistant", "content": last_message.content, "tool_calls": tool_calls})
 if tool_calls:
 print(f"\n Iteration {i+1}: Tool calls detected")
 # Execute tool calls
 for call in tool_calls:
 func_name = call.function.name
 args = json.loads(call.function.arguments)
 # Print tool name and arguments
 print(f" Calling tool: {func_name} with arguments: {args}")
 result = tools_by_name[func_name](**args)
 # Append tool response
 messages.append({
 "role": "tool",
 "tool_call_id": call.id,
 "content": str(result)
 })
 else:
 # No tool calls → final answer
 print(f"\nFinal answer after {i+1} iterations: {last_message.content}")
 return last_message.content
 return "Max iterations reached without a final answer."
# --- Run ---
answer = run_agent("Please multiply 3 and 4 and divide the result by 2")

Conclusion

I hope these examples have helped the reader realise the simplicity behind the patterns for implementing agentic workflows. While using higher-level packages like LangChain offers ‘shorter’ implementation methods, this often obscures the underlying details. To truly understand these core concepts, I believe a simpler approach using just the API and basic Python is most effective.

LLM are way too confident

Calvin Tran — Sat, 11 Oct 2025 00:00:00 +0000

The Paradox of High Accuracy and Low Reliability

In this article, we’ll explore LLM’s impressive ability to achieve high zero-shot performance on a classification tasks and at the same time being way too overconfident on all of their responses.

Calibration it ensures that the model “knows what it doesn’t know.” For instance, if a model assigns a probability of 70% to a collection of predictions, we expect that 70% of those specific predictions will be correct in reality. When a model is perfectly calibrated, you can directly trust its confidence scores as an honest statement of its likelihood of being right.

GPT can easily classify a sample without any prior fine-tuning but its confidence scores often don’t accurately reflect the likelihood of its predictions being correct. This disparity between performance and reliability is a significant challenge for deploying these models in real-world applications.

The data

The dataset for this article is sourced from scikit-learn (sklearn) open datasets. This is a classic NLP classification problem where the raw data consists of various news articles, and the target variable maps to 20 distinct article topics.

For the sake of simplifying this demonstration, we filtered the original corpus. Our reduced dataset now exclusively comprises articles belonging to three specific categories: ‘mac’, ‘motorcycles’, and ‘baseball’.

from sklearn.datasets import fetch_20newsgroups

categories_to_fetch = ['comp.sys.mac.hardware','rec.motorcycles','rec.sport.baseball']
simplified_names = ['mac','motor','baseball']

# Fetch the data, removing headers/footers/quotes for cleaner text
newsgroups_train = fetch_20newsgroups(
 subset='train',
 remove=('headers', 'footers', 'quotes'),
 categories=categories_to_fetch,
 shuffle=True,
 random_state=42
)
newsgroups_test = fetch_20newsgroups(
 subset='test',
 remove=('headers', 'footers', 'quotes'),
 categories=categories_to_fetch,
 shuffle=True,
 random_state=42
)

The baseline model

Before the AI model, we first need to establish a baseline classifier. This was accomplished using a classical machine learning approach: we vectorized the text data using the well-established TF-IDF (Term Frequency-Inverse Document Frequency) method, and then trained a simple Naive Bayes classifier on the resulting features.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
vectorizer = TfidfVectorizer()
vectors = vectorizer.fit_transform(newsgroups_train.data)
clf = MultinomialNB(alpha=.01)
clf.fit(vectors, newsgroups_train.target)
vectors_test = vectorizer.transform(newsgroups_test.data)
pred = clf.predict(vectors_test)

Our Naive Bayes baseline model achieved a respectable 85% F1 score. Crucially, it also demonstrated a decently calibrated output:

The AI model

Now we move to a modern approach for this NLP task: zero-shot classification. By leveraging the OpenAI API we completely bypass the need for any training. Instead, we tap directly into the vast pre-training knowledge of GPT and use a prompt to instruct the model to classify the news articles. This method allows us to achieve high performance without a single training loop.

This Python snippet defines a function, get_api_metrics_data, which performs zero-shot classification on a given news article using the OpenAI GPT-4o-mini model. It uses a system prompt to enforce classification into one of three specified categories (simplified_names) and is configured to return log probabilities for the top tokens (logprobs=True, top_logprobs=3) OpenAI LogProb cookbook.

from openai import OpenAI

# Prompt 
SYSTEM_PROMPT = f"""You are an expert classifier.
Classify the article into exactly one of the following categories: {', '.join(simplified_names)}.
Return ONLY the category name, exactly as it appears in the list, and nothing else."""

# Inference function
def get_api_metrics_data(article):
 """
 Calls the API once and returns both the probability vector (for calibration)
 and the predicted integer ID.
 """
 prob_vector = np.zeros(3)
 pred_id = -1

 try:
 completion = client.chat.completions.create(
 model="gpt-4o-mini",
 messages=[{"role": "system", "content": SYSTEM_PROMPT},
 {"role": "user", "content": f"Article: {article}...\nCategory:"}],
 logprobs=True,
 top_logprobs=3,
 temperature=0,
 )

 predicted_name = completion.choices[0].message.content.strip().lower()
 pred_id = category_to_id.get(predicted_name, -1)
 logprobs_content = completion.choices[0].logprobs.content
 if logprobs_content:
 first_token_logprobs = logprobs_content[0].top_logprobs
 class_probs = np.zeros(3)
 for lp_item in first_token_logprobs:
 token = lp_item.token.strip().lower()
 linear_prob = np.exp(lp_item.logprob)
 if 'mac' in token or token == 'mac':
 class_probs[0] += linear_prob
 elif 'motor' in token or token == 'motor':
 class_probs[1] += linear_prob
 elif 'baseball' in token or token == 'baseball':
 class_probs[2] += linear_prob
 total_relevant_prob = np.sum(class_probs)
 if total_relevant_prob > 0:
 prob_vector = class_probs / total_relevant_prob
 else:
 if pred_id != -1:
 prob_vector[pred_id] = 1.0
 return prob_vector, pred_id
 except Exception:
 return np.zeros(3), -1

Our AI model demonstrated an impressive F1 score of 95%, significantly outperforming the Naive Bayes baseline by a solid 10% margin. However the model exhibits severe miscalibration, oscillating between being overconfident when its predictions are likely wrong, and being overly pessimistic (underconfident) when its predictions are correct :

Can we fix it ?

To address the observed miscalibration, we applied a traditional post-processing technique: fitting a second regression model to map the estimated scores to true probabilities. Specifically, we leveraged the Isotonic regression from the scikit-learn package, a common method for recalibrating uncalibrated estimators.

However, isolating the results for a specific class, such as the ‘hardware’ (mac) category, clearly shows the limitations of this approach:

As the initial confidence scores from the GPT model exhibited virtually no correlation with the real class probabilities, it proved impossible to fix the miscalibration effectively using Isotonic Regression or any other simple linear or non-linear transformation. The underlying issue is the fundamental instability of the model’s confidence scores.

Why is this a problem ?

In most practical applications, simply providing a prediction label is insufficient; a clear measure of the confidence level in that prediction is absolutely essential. Consider a fraud detection system: it is designed to automatically approve transactions with high-confidence predictions of safety, while low-confidence predictions are deliberately flagged for mandatory human review. If the model’s underlying confidence scores are inaccurate or “miscalibrated,” this entire risk-managed workflow completely breaks down, leading to either high-risk transactions being overlooked or excessive manual review of safe transactions.

Conclusion

While Large Language Models (LLMs) like GPT offer incredible zero-shot performance on NLP tasks, completely eliminating the need for training data, they suffer from a fundamental practical limitation: they currently lack an effective and reliable mechanism for measuring output uncertainty and calibration.

This absence means that in any use case where decision-making critically depends on a trustworthy measure of confidence or risk, modern LLMs become a terrible choice. Instead, practitioners should rely on simpler, well-calibrated Machine Learning models, even though these require the initial investment of collecting and training on labeled data. The superior performance of the LLM isn’t always worth the cost of unreliable risk assessment.

Diarization & Transcription with Whisper and PyAnnote

Calvin Tran — Fri, 06 Jun 2025 00:00:00 +0000

Speaker Diarization answers the question “who spoke when?” by segmenting an audio stream based on speaker identity, while transcription tells us “what was said”.

Introducing the Key Components

Whisper: Automatic Speech Recognition model that converts speech to text.
Pyannote.audio: Deep neural network for speaker diarization.
WhisperX: Optimized Whisper variant that integrates multiple components.

A Step-by-Step Implementation Guide

This section provides a practical guide to implementing speaker diarization using WhisperX, Pyannote, and Whisper, while highlighting WhisperX’s advantages. For our audio sample, we can grab a segment from pretty much any free podcast out there – let’s go for a clip from Lex Friedman, one of my personal favorites.

Setup

Before diving into the code, it is essential to prepare your development environment. Make sure you have a Python virtual environment set up, it would also be beneficial to have access to a GPU compatible with CUDA for acceleration.

python -m venv venv
source venv/bin/activate

To utilize Pyannote.audio models for diarization through WhisperX, you will need a Hugging Face access token with ‘read’ permissions, for which you must accept the user agreements for the pyannote/segmentation-3.0 and pyannote/speaker-diarization-3.1 models on the Hugging Face website: HuggingFace

Finally we can pip install our dependencies:

python -m pip install torch numpy pyannote.audio whisper whisperx

First, we’ll import the necessary libraries, and configure our Hugging Face token and file path.

import torch
import whisper
from pyannote.audio import Pipeline
from whisperx.diarize import DiarizationPipeline,assign_word_speakers
from whisperx import load_align_model, align
hugging_face_token="hf_token"
audio_path="audio_file_path"
# Check if cuda is available
import torch
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
DEVICE

Diarization

In the first code snippet, a pre-trained pyannote model is utilized to determine when different speakers are active in an audio recording. The output of this speaker segmentation can be viewed via the print statement:

#Initialize pipeline
pipeline = Pipeline.from_pretrained(
 "pyannote/speaker-diarization-3.1",
 use_auth_token=hugging_face_token)
# send pipeline to GPU (when available)
pipeline.to(torch.device("cuda"))
# apply pretrained pipeline
diarization = pipeline(audio_path)
# print the result
for turn, _, speaker in diarization.itertracks(yield_label=True):
 print(f"start={turn.start:.1f}s stop={turn.end:.1f}s speaker_{speaker}")

This plot illustrates the various segments identified by the model throughout the podcast’s duration.

Transcription

Now let’s run the Whisper model to generate the transcription:

model_name = "turbo"
model = whisper.load_model(model_name, DEVICE)
script = model.transcribe(audio_path)

Naive alignment

The following script attempts a basic diarization-to-transcription alignment. It directly links diarization segments (which identify who spoke when) with Whisper’s transcription segments (which capture what was said) by simply checking if their timestamps overlap. However, this straightforward timestamp-based matching is often insufficient and can lead to significant inaccuracies, such as misaligned text, missing words, or incorrect speaker attribution, because it doesn’t account for precise word timings or subtle overlaps in speech.

# Create a text associating diarization segments with Whisper transcription
output_text = []
for turn, _, speaker in diarization.itertracks(yield_label=True):
 segment_text = []
 for segment in script["segments"]:
 if segment["start"] >= turn.start and segment["end"] <= turn.end:
 segment_text.append(segment["text"])
 combined_text = " ".join(segment_text)
 output_text.append(f"{speaker} [{turn.start:.2f}s - {turn.end:.2f}s]: {combined_text}")

# Print the result
for line in output_text:
 print(line)

Advanced alignment - WhisperX

While a simple timestamp overlap can lead to significant inaccuracies in aligning diarization and transcription segments, WhisperX addresses these challenges through forced alignment. Unlike basic timestamp matching, WhisperX refines the alignment by first leveraging the more precise word-level timestamps generated by its underlying Whisper model. It then performs a forced alignment process, which essentially “snaps” the diarization segments to the exact start and end times of the individual words within the transcription. This method accounts for subtle overlaps and ensures that each word is attributed to the correct speaker, even when speech segments are close together or overlap slightly.

This script first uses the pyannote library to identify speakers and time of enunciation in an audio file (that’s the “diarization” part). Then, it takes a pre-generated text transcript (script[“segments”]) and uses WhisperX’s alignment models to precisely match each word in the transcript to its exact timing in the audio. Finally, it combines the speaker information from pyannote with the word-level timings from WhisperX to produce a highly accurate, time-stamped transcript where each spoken word is correctly attributed to the right speaker.

# Initialize a diarization pipeline
diarization_pipeline = DiarizationPipeline(use_auth_token=hugging_face_token)
diarized = diarization_pipeline(audio_path)
# Align Script
model_a, metadata = load_align_model(language_code=script["language"], device=DEVICE)
script_aligned = align(script["segments"], model_a, metadata, audio_path, DEVICE)
# Align Speakers
result_segments, word_seg = list(assign_word_speakers(
 diarized, script_aligned
).values())
transcribed = []
for result_segment in result_segments:
 transcribed.append(
 {
 "start": result_segment["start"],
 "end": result_segment["end"],
 "text": result_segment["text"],
 "speaker": result_segment["speaker"],
 }
 )
for start, end, text, speaker in [i.values() for i in transcribed]:
 print(start, end, speaker, text)

By combining Pyannote’s precise speaker diarization with WhisperX’s advanced forced alignment, this guide demonstrates how to generate highly accurate, speaker-attributed transcripts. This approach significantly surpasses basic timestamp matching, ensuring precise information about “who spoke when and what was said”.

Francesco Gabellini

Is Jev calibrated?

1. Data

2. One Jev call

3. Classify the sample

4. Macro F1

5. Calibration plot (per class, one-vs-rest)

6. Expected Calibration Error

7. Platt recalibration

8. Comparison with the OpenAI run

Classification is back

The decision was always the product

What automation actually asks for

The hoops

System One models

Calibration is king

The part I am not sold on yet

Conclusion

Who Watches the Watchmen?

The old world: NLP measures

Closing the gap: Natural Language Inference

The new world : LLM-as-Judge

Back to basics: the golden dataset

The user as ground truth

A practical framework

Conclusion

References

PageIndex: When Your RAG Reads Like a Human

The Explainability Problem Nobody Talks About

How Humans Actually Look Things Up

Two Phases: Index Then Retrieve

The Code

Stateless by Design, Storable if Needed

Structure-Aware Chunking vs Fixed-Size Chunking

Scaling Limitations

Conclusion

Do Foundational Models Actually Work for Forecasting?

Traditional forecasting

Deep learning did not solve the problem

What TimesFM changes

Running the same backtesting as SARIMA and LightGBM

Why this could democratise forecasting

The limits are real

Conclusion

Building effective agents

Agentic pattern

0) Set-up

1) Prompt chaining

2) Routing

3) Parallelization

4) Evaluator-optimizer

5) Agents

Conclusion

LLM are way too confident

The Paradox of High Accuracy and Low Reliability

The data

The baseline model

The AI model

Can we fix it ?

Why is this a problem ?

Conclusion

Diarization & Transcription with Whisper and PyAnnote

Introducing the Key Components

A Step-by-Step Implementation Guide

Setup

Diarization

Transcription

Naive alignment

Advanced alignment - WhisperX

References :