{"version":"https://jsonfeed.org/version/1","title":"Francesco Gabellini","home_page_url":"http://gabelfra1.github.io/gabelfra.github.io/","feed_url":"http://gabelfra1.github.io/gabelfra.github.io/feed.json","description":"A minimal hugo theme focus on content","favicon":"http://gabelfra1.github.io/gabelfra.github.io//assets/favicon.ico","expired":false,"author":{"name":"Calvin Tran","url":"http://gabelfra1.github.io/gabelfra.github.io/"},"items":[{"id":"53d7043de381fb96b0a7cd6faf834e5eea3b0714","title":"Is Jev calibrated?","summary":"","content_text":"The same experiment as confidence.ipynb (OpenAI gpt-4o-mini + logprobs), redone with TypeSafe\u0026rsquo;s Jev: 20 Newsgroups posts, classified as mac / motor / baseball, then checked with reliability curves and ECE.\nWhat is the same: the data (test split, random_state=42, first 500 posts, truncated to 500 characters), the F1 score, the per-class reliability curves, and ECE.\nWhat is different:\nJev\u0026rsquo;s Choice answer returns a probability for every option, so there is no logprob-token heuristic to map tokens back to classes. The API key is read from TYPESAFE_API_KEY (or prompted for), never hardcoded. The ECE function is fixed: the original silently dropped every prediction with probability exactly 1.0 (the most overconfident ones). Recalibration uses Platt scaling, scored out-of-fold, and the last section compares Jev (raw and recalibrated) with the OpenAI run. All analysis uses Jev\u0026rsquo;s probabilities. Its confidence field is a rescaled peak of that distribution, not a probability of being correct, so it is saved but not used for calibration.\nimport os, sys, subprocess, getpass try: import typesafe_sdk except ModuleNotFoundError: subprocess.check_call([sys.executable, \u0026#34;-m\u0026#34;, \u0026#34;pip\u0026#34;, \u0026#34;install\u0026#34;, \u0026#34;-q\u0026#34;, \u0026#34;typesafe-sdk\u0026#34;]) # pip may upgrade packages this kernel already imported (e.g. typing_extensions), so a restart is required raise RuntimeError(\u0026#34;typesafe-sdk was just installed. Restart the kernel (Kernel \u0026gt; Restart) and run all cells again.\u0026#34;) if not os.environ.get(\u0026#34;TYPESAFE_API_KEY\u0026#34;): os.environ[\u0026#34;TYPESAFE_API_KEY\u0026#34;] = getpass.getpass(\u0026#34;TypeSafe API key (create one at https://console.typesafe.ai/): \u0026#34;) import numpy as np import pandas as pd import matplotlib.pyplot as plt from concurrent.futures import ThreadPoolExecutor, as_completed from sklearn import metrics from sklearn.calibration import calibration_curve from sklearn.datasets import fetch_20newsgroups from sklearn.linear_model import LogisticRegression from sklearn.model_selection import StratifiedKFold from typesafe_sdk import Choice, TypeSafeClient, TypeSafeError 1. Data MODEL = \u0026#34;jev-latest\u0026#34; SAMPLE_SIZE = 500 # same as the OpenAI notebook MAX_CHARS = 500 # same truncation as the OpenAI notebook MAX_WORKERS = 8 # parallel requests; the SDK retries 429s with backoff RESULTS_CSV = \u0026#34;jev_classification_results.csv\u0026#34; FORCE_RECOMPUTE = False # True = ignore the cached CSV and call the API again categories_to_fetch = [\u0026#39;comp.sys.mac.hardware\u0026#39;, \u0026#39;rec.motorcycles\u0026#39;, \u0026#39;rec.sport.baseball\u0026#39;] simplified_names = [\u0026#39;mac\u0026#39;, \u0026#39;motor\u0026#39;, \u0026#39;baseball\u0026#39;] prob_cols = [f\u0026#34;prob_{n}\u0026#34; for n in simplified_names] fetch_kwargs = dict(remove=(\u0026#39;headers\u0026#39;, \u0026#39;footers\u0026#39;, \u0026#39;quotes\u0026#39;), categories=categories_to_fetch, shuffle=True, random_state=42) newsgroups_train = fetch_20newsgroups(subset=\u0026#39;train\u0026#39;, **fetch_kwargs) newsgroups_test = fetch_20newsgroups(subset=\u0026#39;test\u0026#39;, **fetch_kwargs) assert list(newsgroups_test.target_names) == categories_to_fetch # so target id i \u0026lt;-\u0026gt; simplified_names[i] sample_data = newsgroups_test.data[:SAMPLE_SIZE] y_true_sample = newsgroups_test.target[:SAMPLE_SIZE] print(f\u0026#34;{len(sample_data)} test articles; class counts: {np.bincount(y_true_sample).tolist()}\u0026#34;) 500 test articles; class counts: [163, 176, 161] 2. One Jev call One Choice question. The option names are what the answer is keyed by, and their descriptions are sent to the model, so each one says what the newsgroup covers.\nQUESTIONS = { \u0026#34;category\u0026#34;: Choice( instructions=\u0026#34;Which newsgroup was this post written in?\u0026#34;, criteria={ \u0026#34;mac\u0026#34;: \u0026#34;Apple Macintosh computer hardware\u0026#34;, \u0026#34;motor\u0026#34;: \u0026#34;Motorcycles\u0026#34;, \u0026#34;baseball\u0026#34;: \u0026#34;Baseball\u0026#34;, }, ) } def classify(client, article): \u0026#34;\u0026#34;\u0026#34;One Jev call -\u0026gt; (probability vector in simplified_names order, Jev confidence, model version).\u0026#34;\u0026#34;\u0026#34; response = client.system_one(article[:MAX_CHARS], QUESTIONS, model=MODEL) answer = response.choices[\u0026#34;category\u0026#34;] probs = np.array([answer.probabilities.get(n, 0.0) for n in simplified_names]) return probs, answer.confidence, response.model sample_article = newsgroups_train.data[20] true_category = simplified_names[newsgroups_train.target[20]] with TypeSafeClient() as client: response = client.system_one(sample_article[:MAX_CHARS], QUESTIONS, model=MODEL) answer = response.choices[\u0026#34;category\u0026#34;] print(f\u0026#34;True category: {true_category}\u0026#34;) print(f\u0026#34;Jev choice: {answer.choice} (confidence {answer.confidence:.3f}, model {response.model})\u0026#34;) display(pd.Series(answer.probabilities, name=\u0026#34;probability\u0026#34;).sort_values(ascending=False).to_frame()) 3. Classify the sample Results are written to jev_classification_results.csv as soon as the calls finish, and reused on later runs (set FORCE_RECOMPUTE = True to redo them). Failed calls are counted and excluded, like the invalid predictions in the OpenAI notebook.\ndef run_one(client, i, article): try: probs, confidence, model = classify(client, article) return i, probs, confidence, model, None except TypeSafeError as e: return i, None, None, None, repr(e) if os.path.exists(RESULTS_CSV) and not FORCE_RECOMPUTE: print(f\u0026#34;Using cached results from {RESULTS_CSV}\u0026#34;) else: rows, errors = [], {} print(f\u0026#34;Starting Jev calls for {len(sample_data)} articles...\u0026#34;) with TypeSafeClient() as client, ThreadPoolExecutor(MAX_WORKERS) as pool: futures = [pool.submit(run_one, client, i, a) for i, a in enumerate(sample_data)] for done, future in enumerate(as_completed(futures), 1): i, probs, confidence, model, error = future.result() if error: errors[i] = error else: rows.append({\u0026#34;article_idx\u0026#34;: i, \u0026#34;y_true_final\u0026#34;: int(y_true_sample[i]), **dict(zip(prob_cols, probs)), \u0026#34;confidence\u0026#34;: confidence, \u0026#34;model\u0026#34;: model}) if done % 50 == 0: print(f\u0026#34;Processed {done}/{len(sample_data)} articles.\u0026#34;) if not rows: raise RuntimeError(f\u0026#34;Every Jev call failed. First error: {next(iter(errors.values()))}\u0026#34;) pd.DataFrame(rows).sort_values(\u0026#34;article_idx\u0026#34;).to_csv(RESULTS_CSV, index=False) print(f\u0026#34;Saved {len(rows)} results to {RESULTS_CSV}; {len(errors)} calls failed.\u0026#34;) if errors: print(\u0026#34;First errors:\u0026#34;, list(errors.values())[:3]) results_df = pd.read_csv(RESULTS_CSV) print(f\u0026#34;Model version(s): {results_df[\u0026#39;model\u0026#39;].unique().tolist()}\u0026#34;) Using cached results from jev_classification_results.csv Model version(s): [\u0026#39;jev-1.13.0\u0026#39;] 4. Macro F1 y_true = results_df[\u0026#34;y_true_final\u0026#34;].to_numpy() P = results_df[prob_cols].to_numpy() # (n_samples, 3) probabilities straight from Jev y_pred = P.argmax(axis=1) f1_jev = metrics.f1_score(y_true, y_pred, average=\u0026#34;macro\u0026#34;) print(\u0026#34;==================================================\u0026#34;) print(f\u0026#34;Macro F1 (Jev): {f1_jev:.4f} accuracy: {(y_pred == y_true).mean():.4f}\u0026#34;) print(f\u0026#34;(Calculated on {len(y_true)} valid predictions out of {SAMPLE_SIZE})\u0026#34;) print(\u0026#34;==================================================\u0026#34;) ================================================== Macro F1 (Jev): 0.9186 accuracy: 0.9180 (Calculated on 500 valid predictions out of 500) ================================================== 5. Calibration plot (per class, one-vs-rest) fig, ax = plt.subplots(figsize=(10, 8)) ax.plot([0, 1], [0, 1], \u0026#34;k:\u0026#34;, label=\u0026#34;Perfectly calibrated\u0026#34;) for i, name in enumerate(simplified_names): fraction_of_positives, mean_predicted_value = calibration_curve(y_true == i, P[:, i], n_bins=10) ax.plot(mean_predicted_value, fraction_of_positives, \u0026#34;o-\u0026#34;, label=name) ax.set_xlabel(\u0026#34;Mean Predicted Probability (Jev)\u0026#34;) ax.set_ylabel(\u0026#34;Fraction of Positives (True Probability)\u0026#34;) ax.set_title(f\u0026#34;Calibration Plot (Reliability Curve) for Jev Predictions (N={len(y_true)})\u0026#34;) ax.legend(loc=\u0026#34;lower right\u0026#34;) ax.grid(True, linestyle=\u0026#34;--\u0026#34;, alpha=0.7) plt.savefig(\u0026#34;jev_calibration_plot.png\u0026#34;) plt.show() 6. Expected Calibration Error Same definition as the original notebook, with one fix: np.digitize puts a probability of exactly 1.0 in bin index 10, which the loop never visits, so those predictions were dropped from the average. Jev (like the OpenAI mapping) returns many exact 0.0/1.0 values, and the 1.0 ones are where overconfidence shows up, so they have to be counted.\ndef expected_calibration_error(y_true, prob_pred, n_bins=10): y_true, prob_pred = np.asarray(y_true), np.asarray(prob_pred) bins = np.linspace(0, 1, n_bins + 1) binids = np.clip(np.digitize(prob_pred, bins) - 1, 0, n_bins - 1) # p == 1.0 goes in the last bin ece = 0.0 for b in range(n_bins): mask = binids == b if mask.any(): ece += abs(y_true[mask].mean() - prob_pred[mask].mean()) * mask.sum() / len(y_true) return ece for i, name in enumerate(simplified_names): ece = expected_calibration_error((y_true == i).astype(int), P[:, i]) print(f\u0026#34;ECE for \u0026#39;{name}\u0026#39;: {ece * 100:.2f}%\u0026#34;) ECE for \u0026#39;mac\u0026#39;: 5.34% ECE for \u0026#39;motor\u0026#39;: 4.50% ECE for \u0026#39;baseball\u0026#39;: 1.96% Helper used by the tables below. Top-label ECE asks whether the probability given to the predicted class matches how often that prediction is right (with a bootstrap 95% interval). Per-class ECE alone looks good when most probabilities are near 0, and ECE is biased upward at a few hundred samples, so read the interval before calling a difference real. acc p\u0026gt;=0.99 is the direct test of \u0026ldquo;a probability of 1.0 should be right every time\u0026rdquo;.\nN_CLASSES = len(simplified_names) def summarize(name, y, P, n_boot=1000, seed=0): y_pred, top_prob = P.argmax(axis=1), P.max(axis=1) correct = (y_pred == y).astype(int) rng = np.random.default_rng(seed) boots = [expected_calibration_error(correct[idx], top_prob[idx]) for idx in (rng.integers(0, len(y), len(y)) for _ in range(n_boot))] lo, hi = np.percentile(boots, [2.5, 97.5]) * 100 sure = top_prob \u0026gt;= 0.99 row = { \u0026#34;model\u0026#34;: name, \u0026#34;n\u0026#34;: len(y), \u0026#34;accuracy\u0026#34;: correct.mean(), \u0026#34;macro F1\u0026#34;: metrics.f1_score(y, y_pred, average=\u0026#34;macro\u0026#34;), \u0026#34;mean top prob\u0026#34;: top_prob.mean(), \u0026#34;top-label ECE %\u0026#34;: expected_calibration_error(correct, top_prob) * 100, \u0026#34;95% CI\u0026#34;: f\u0026#34;[{lo:.1f}, {hi:.1f}]\u0026#34;, \u0026#34;Brier\u0026#34;: np.mean(np.sum((P - np.eye(P.shape[1])[y]) ** 2, axis=1)), \u0026#34;n p\u0026gt;=0.99\u0026#34;: int(sure.sum()), \u0026#34;acc p\u0026gt;=0.99\u0026#34;: correct[sure].mean() if sure.any() else np.nan, } for i, c in enumerate(simplified_names): row[f\u0026#34;ECE {c} %\u0026#34;] = expected_calibration_error((y == i).astype(int), P[:, i]) * 100 return row 7. Platt recalibration Platt scaling fits one sigmoid per class on the logit of Jev\u0026rsquo;s probability (2 parameters per class), then renormalises the rows to sum to 1.\nIt is scored out-of-fold: with 5-fold cross-validation each article is recalibrated by a model that never saw it, so all ~500 articles count and nothing is scored on the data it was fitted on. Probabilities of exactly 0 or 1 are clipped to 1e-6 first, because the logit of 0 or 1 is infinite.\nEPS = 1e-6 def _logit(p): p = np.clip(p, EPS, 1 - EPS) return np.log(p / (1 - p)) def fit_platt(P_fit, y_fit): return [LogisticRegression(C=1e3).fit(_logit(P_fit[:, [k]]), (y_fit == k).astype(int)) for k in range(P_fit.shape[1])] def apply_platt(models, P_new): cal = np.column_stack([m.predict_proba(_logit(P_new[:, [k]]))[:, 1] for k, m in enumerate(models)]) totals = cal.sum(axis=1, keepdims=True) return np.where(totals \u0026gt; 0, cal / np.where(totals \u0026gt; 0, totals, 1), 1 / cal.shape[1]) def out_of_fold_platt(P, y, n_splits=5, seed=42): out = np.zeros_like(P) for fit_idx, test_idx in StratifiedKFold(n_splits, shuffle=True, random_state=seed).split(P, y): out[test_idx] = apply_platt(fit_platt(P[fit_idx], y[fit_idx]), P[test_idx]) return out P_platt = out_of_fold_platt(P, y_true) fig, axes = plt.subplots(1, N_CLASSES, figsize=(18, 5), sharey=True) for i, (name, ax) in enumerate(zip(simplified_names, axes)): ax.plot([0, 1], [0, 1], \u0026#34;k:\u0026#34;, label=\u0026#34;Perfectly calibrated\u0026#34;) for probs, style, label in [(P, \u0026#34;o--\u0026#34;, \u0026#34;Jev (raw)\u0026#34;), (P_platt, \u0026#34;s-\u0026#34;, \u0026#34;Jev + Platt (out-of-fold)\u0026#34;)]: frac, mean_p = calibration_curve(y_true == i, probs[:, i], n_bins=10) ax.plot(mean_p, frac, style, label=label, alpha=0.8) ax.set_title(f\u0026#34;\u0026#39;{name}\u0026#39; (N={len(y_true)})\u0026#34;) ax.set_xlabel(\u0026#34;Mean Predicted Probability\u0026#34;) ax.grid(True, linestyle=\u0026#34;--\u0026#34;, alpha=0.7) axes[0].set_ylabel(\u0026#34;Fraction of Positives\u0026#34;) axes[0].legend(loc=\u0026#34;upper left\u0026#34;) plt.tight_layout() plt.savefig(\u0026#34;jev_platt_calibration_plot.png\u0026#34;) plt.show() print(f\u0026#34;Accuracy: raw {(P.argmax(1) == y_true).mean():.4f} | Platt {(P_platt.argmax(1) == y_true).mean():.4f}\u0026#34;) for i, name in enumerate(simplified_names): before = expected_calibration_error((y_true == i).astype(int), P[:, i]) * 100 after = expected_calibration_error((y_true == i).astype(int), P_platt[:, i]) * 100 print(f\u0026#34;ECE \u0026#39;{name}\u0026#39;: {before:.2f}% -\u0026gt; {after:.2f}% after Platt\u0026#34;) Accuracy: raw 0.9180 | Platt 0.9260 ECE \u0026#39;mac\u0026#39;: 5.34% -\u0026gt; 2.23% after Platt ECE \u0026#39;motor\u0026#39;: 4.50% -\u0026gt; 2.10% after Platt ECE \u0026#39;baseball\u0026#39;: 1.96% -\u0026gt; 1.87% after Platt 8. Comparison with the OpenAI run openai_classification_results.csv does not keep the article index (and has 496 valid rows), so the OpenAI run is a different sample of the same distribution: compare it as a reference rather than row by row. Its probabilities come from the first-token logprob heuristic in confidence.ipynb, renormalised over the matched tokens; Jev\u0026rsquo;s are its native probabilities.\nruns = { \u0026#34;Jev\u0026#34;: (y_true, P), \u0026#34;Jev + Platt (out-of-fold)\u0026#34;: (y_true, P_platt), } if os.path.exists(\u0026#34;openai_classification_results.csv\u0026#34;): openai_df = pd.read_csv(\u0026#34;openai_classification_results.csv\u0026#34;) runs[\u0026#34;OpenAI gpt-4o-mini\u0026#34;] = (openai_df[\u0026#34;y_true_final\u0026#34;].to_numpy(), openai_df[prob_cols].to_numpy()) else: print(\u0026#34;openai_classification_results.csv not found; showing Jev only.\u0026#34;) summary = pd.DataFrame([summarize(name, y, probs) for name, (y, probs) in runs.items()]).set_index(\u0026#34;model\u0026#34;) display(summary.round(3)) metric Jev Jev + Platt (out-of-fold) OpenAI gpt-4o-mini n 500 500 496 accuracy 0.918 0.926 0.911 macro F1 0.919 0.926 0.913 mean top prob 0.942 0.920 0.986 top-label ECE % 3.524 1.248 7.454 95% CI [2.5, 5.8] [1.1, 3.4] [5.4, 9.8] Brier 0.115 0.099 0.154 n p\u0026gt;=0.99 356 328 451 acc p\u0026gt;=0.99 0.997 0.997 0.967 ECE mac % 5.338 2.232 6.813 ECE motor % 4.498 2.102 5.224 ECE baseball % 1.962 1.868 3.437 colors = {\u0026#34;Jev\u0026#34;: \u0026#34;tab:blue\u0026#34;, \u0026#34;Jev + Platt (out-of-fold)\u0026#34;: \u0026#34;tab:green\u0026#34;, \u0026#34;OpenAI gpt-4o-mini\u0026#34;: \u0026#34;tab:orange\u0026#34;} fig, axes = plt.subplots(1, N_CLASSES + 1, figsize=(22, 5), sharey=True) for ax, title in zip(axes, [*simplified_names, \u0026#34;top label (all classes)\u0026#34;]): ax.plot([0, 1], [0, 1], \u0026#34;k:\u0026#34;, label=\u0026#34;Perfectly calibrated\u0026#34;) ax.set_title(title) ax.set_xlabel(\u0026#34;Mean Predicted Probability\u0026#34;) ax.grid(True, linestyle=\u0026#34;--\u0026#34;, alpha=0.7) for name, (y, probs) in runs.items(): for i in range(N_CLASSES): frac, mean_p = calibration_curve(y == i, probs[:, i], n_bins=10) axes[i].plot(mean_p, frac, \u0026#34;o-\u0026#34;, color=colors[name], label=name) frac, mean_p = calibration_curve(probs.argmax(1) == y, probs.max(1), n_bins=10) axes[-1].plot(mean_p, frac, \u0026#34;o-\u0026#34;, color=colors[name], label=name) axes[0].set_ylabel(\u0026#34;Fraction of Positives\u0026#34;) axes[0].legend(loc=\u0026#34;upper left\u0026#34;) plt.tight_layout() plt.savefig(\u0026#34;jev_vs_openai_calibration.png\u0026#34;) plt.show() ","content_html":"\u003cp\u003eThe same experiment as \u003ccode\u003econfidence.ipynb\u003c/code\u003e (OpenAI \u003ccode\u003egpt-4o-mini\u003c/code\u003e + logprobs), redone with TypeSafe\u0026rsquo;s \u003cstrong\u003eJev\u003c/strong\u003e:\n20 Newsgroups posts, classified as \u003ccode\u003emac\u003c/code\u003e / \u003ccode\u003emotor\u003c/code\u003e / \u003ccode\u003ebaseball\u003c/code\u003e, then checked with reliability curves and ECE.\u003c/p\u003e\n\u003cp\u003eWhat is the same: the data (test split, \u003ccode\u003erandom_state=42\u003c/code\u003e, first 500 posts, truncated to 500 characters), the F1 score, the per-class reliability curves, and ECE.\u003c/p\u003e\n\u003cp\u003eWhat is different:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eJev\u0026rsquo;s \u003cstrong\u003eChoice\u003c/strong\u003e answer returns a probability for every option, so there is no logprob-token heuristic to map tokens back to classes.\u003c/li\u003e\n\u003cli\u003eThe API key is read from \u003ccode\u003eTYPESAFE_API_KEY\u003c/code\u003e (or prompted for), never hardcoded.\u003c/li\u003e\n\u003cli\u003eThe ECE function is fixed: the original silently dropped every prediction with probability exactly \u003ccode\u003e1.0\u003c/code\u003e (the most overconfident ones).\u003c/li\u003e\n\u003cli\u003eRecalibration uses \u003cstrong\u003ePlatt scaling\u003c/strong\u003e, scored out-of-fold, and the last section compares Jev (raw and recalibrated) with the OpenAI run.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAll analysis uses Jev\u0026rsquo;s \u003ccode\u003eprobabilities\u003c/code\u003e. Its \u003ccode\u003econfidence\u003c/code\u003e field is a rescaled peak of that distribution, not a probability of being correct, so it is saved but not used for calibration.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eos\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esys\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esubprocess\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003egetpass\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003etry\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etypesafe_sdk\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003eexcept\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eModuleNotFoundError\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003esubprocess\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echeck_call\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e([\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esys\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eexecutable\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;-m\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;pip\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;install\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;-q\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;typesafe-sdk\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#75715e\"\u003e# pip may upgrade packages this kernel already imported (e.g. typing_extensions), so a restart is required\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003eraise\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eRuntimeError\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;typesafe-sdk was just installed. Restart the kernel (Kernel \u0026gt; Restart) and run all cells again.\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003enot\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eos\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eenviron\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eget\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;TYPESAFE_API_KEY\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eos\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eenviron\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;TYPESAFE_API_KEY\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003egetpass\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003egetpass\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;TypeSafe API key (create one at https://console.typesafe.ai/): \u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enumpy\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eas\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003epandas\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eas\u003c/span\u003e \u003cspan style=\"color:#111\"\u003epd\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ematplotlib.pyplot\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eas\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eplt\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003econcurrent.futures\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eThreadPoolExecutor\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eas_completed\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esklearn\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emetrics\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esklearn.calibration\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecalibration_curve\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esklearn.datasets\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efetch_20newsgroups\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esklearn.linear_model\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eLogisticRegression\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esklearn.model_selection\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eStratifiedKFold\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etypesafe_sdk\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eChoice\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eTypeSafeClient\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eTypeSafeError\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"1-data\"\u003e1. Data\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eMODEL\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;jev-latest\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eSAMPLE_SIZE\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e500\u003c/span\u003e          \u003cspan style=\"color:#75715e\"\u003e# same as the OpenAI notebook\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eMAX_CHARS\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e500\u003c/span\u003e            \u003cspan style=\"color:#75715e\"\u003e# same truncation as the OpenAI notebook\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eMAX_WORKERS\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e8\u003c/span\u003e            \u003cspan style=\"color:#75715e\"\u003e# parallel requests; the SDK retries 429s with backoff\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eRESULTS_CSV\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;jev_classification_results.csv\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eFORCE_RECOMPUTE\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eFalse\u003c/span\u003e    \u003cspan style=\"color:#75715e\"\u003e# True = ignore the cached CSV and call the API again\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003ecategories_to_fetch\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;comp.sys.mac.hardware\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#39;rec.motorcycles\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#39;rec.sport.baseball\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003esimplified_names\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;mac\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#39;motor\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#39;baseball\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eprob_cols\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;prob_\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003en\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003en\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esimplified_names\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003efetch_kwargs\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edict\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eremove\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;headers\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#39;footers\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#39;quotes\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecategories\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecategories_to_fetch\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eshuffle\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eTrue\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erandom_state\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e42\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003enewsgroups_train\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efetch_20newsgroups\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esubset\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;train\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e**\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efetch_kwargs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003enewsgroups_test\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efetch_20newsgroups\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esubset\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;test\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e**\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efetch_kwargs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003eassert\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elist\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003enewsgroups_test\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etarget_names\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecategories_to_fetch\u003c/span\u003e  \u003cspan style=\"color:#75715e\"\u003e# so target id i \u0026lt;-\u0026gt; simplified_names[i]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003esample_data\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enewsgroups_test\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edata\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[:\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eSAMPLE_SIZE\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003ey_true_sample\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enewsgroups_test\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etarget\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[:\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eSAMPLE_SIZE\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elen\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esample_data\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e test articles; class counts: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ebincount\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true_sample\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etolist\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e500 test articles; class counts: [163, 176, 161]\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"2-one-jev-call\"\u003e2. One Jev call\u003c/h3\u003e\n\u003cp\u003eOne \u003ccode\u003eChoice\u003c/code\u003e question. The option names are what the answer is keyed by, and their descriptions are sent to the model, so each one says what the newsgroup covers.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eQUESTIONS\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;category\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eChoice\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003einstructions\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Which newsgroup was this post written in?\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003ecriteria\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;mac\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Apple Macintosh computer hardware\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;motor\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Motorcycles\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;baseball\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Baseball\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eclassify\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003earticle\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u0026#34;One Jev call -\u0026gt; (probability vector in simplified_names order, Jev confidence, model version).\u0026#34;\u0026#34;\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esystem_one\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003earticle\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[:\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eMAX_CHARS\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eQUESTIONS\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eMODEL\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eanswer\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoices\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;category\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eprobs\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003earray\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e([\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eanswer\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eprobabilities\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eget\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003en\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e0.0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003en\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esimplified_names\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprobs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eanswer\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econfidence\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003esample_article\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enewsgroups_train\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edata\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e20\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003etrue_category\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esimplified_names\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003enewsgroups_train\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etarget\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e20\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003ewith\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eTypeSafeClient\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eas\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esystem_one\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esample_article\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[:\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eMAX_CHARS\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eQUESTIONS\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eMODEL\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eanswer\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoices\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;category\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;True category: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etrue_category\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Jev choice:    \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eanswer\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoice\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e  (confidence \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eanswer\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econfidence\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e:\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e.3f\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e, model \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e)\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003edisplay\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003epd\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eSeries\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eanswer\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eprobabilities\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;probability\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esort_values\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eascending\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eFalse\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eto_frame\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"3-classify-the-sample\"\u003e3. Classify the sample\u003c/h3\u003e\n\u003cp\u003eResults are written to \u003ccode\u003ejev_classification_results.csv\u003c/code\u003e as soon as the calls finish, and reused on later runs (set \u003ccode\u003eFORCE_RECOMPUTE = True\u003c/code\u003e to redo them). Failed calls are counted and excluded, like the invalid predictions in the OpenAI notebook.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003erun_one\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003earticle\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003etry\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003eprobs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003econfidence\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclassify\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003earticle\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprobs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003econfidence\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eNone\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003eexcept\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eTypeSafeError\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eas\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ee\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eNone\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eNone\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eNone\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erepr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ee\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eos\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003epath\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eexists\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eRESULTS_CSV\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eand\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003enot\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eFORCE_RECOMPUTE\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Using cached results from \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eRESULTS_CSV\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003eelse\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003erows\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eerrors\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Starting Jev calls for \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elen\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esample_data\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e articles...\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ewith\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eTypeSafeClient\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eas\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eThreadPoolExecutor\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eMAX_WORKERS\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eas\u003c/span\u003e \u003cspan style=\"color:#111\"\u003epool\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003efutures\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003epool\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esubmit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003erun_one\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ea\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ea\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eenumerate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esample_data\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edone\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efuture\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eenumerate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eas_completed\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efutures\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprobs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003econfidence\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eerror\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efuture\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eresult\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eerror\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#111\"\u003eerrors\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eerror\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#00a8c8\"\u003eelse\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#111\"\u003erows\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eappend\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e({\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;article_idx\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;y_true_final\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true_sample\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                             \u003cspan style=\"color:#f92672\"\u003e**\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edict\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ezip\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eprob_cols\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprobs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)),\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;confidence\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003econfidence\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;model\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e})\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edone\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e%\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e50\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Processed \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edone\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e/\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elen\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esample_data\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e articles.\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003enot\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erows\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003eraise\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eRuntimeError\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Every Jev call failed. First error: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003enext\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eiter\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eerrors\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003evalues\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()))\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003epd\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eDataFrame\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003erows\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esort_values\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;article_idx\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eto_csv\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eRESULTS_CSV\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eindex\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eFalse\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Saved \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elen\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003erows\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e results to \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eRESULTS_CSV\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e; \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elen\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eerrors\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e calls failed.\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eerrors\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;First errors:\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elist\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eerrors\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003evalues\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e())[:\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e3\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eresults_df\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003epd\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eread_csv\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eRESULTS_CSV\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Model version(s): \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eresults_df\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;model\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eunique\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etolist\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eUsing cached results from jev_classification_results.csv\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eModel version(s): [\u0026#39;jev-1.13.0\u0026#39;]\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"4-macro-f1\"\u003e4. Macro F1\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresults_df\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;y_true_final\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eto_numpy\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresults_df\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eprob_cols\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eto_numpy\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e          \u003cspan style=\"color:#75715e\"\u003e# (n_samples, 3) probabilities straight from Jev\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003ey_pred\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eargmax\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eaxis\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003ef1_jev\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emetrics\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ef1_score\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ey_pred\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eaverage\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;macro\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;==================================================\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Macro F1 (Jev): \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ef1_jev\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e:\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e.4f\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e   accuracy: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_pred\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emean\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e:\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e.4f\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;(Calculated on \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elen\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e valid predictions out of \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eSAMPLE_SIZE\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e)\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;==================================================\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e==================================================\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eMacro F1 (Jev): 0.9186   accuracy: 0.9180\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e(Calculated on 500 valid predictions out of 500)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e==================================================\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"5-calibration-plot-per-class-one-vs-rest\"\u003e5. Calibration plot (per class, one-vs-rest)\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003efig\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eplt\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esubplots\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efigsize\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e10\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e8\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eplot\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e([\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;k:\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elabel\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Perfectly calibrated\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eenumerate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esimplified_names\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003efraction_of_positives\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emean_predicted_value\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecalibration_curve\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[:,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003en_bins\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e10\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eplot\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emean_predicted_value\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efraction_of_positives\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;o-\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elabel\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eset_xlabel\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Mean Predicted Probability (Jev)\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eset_ylabel\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Fraction of Positives (True Probability)\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eset_title\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Calibration Plot (Reliability Curve) for Jev Predictions (N=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elen\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e)\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elegend\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eloc\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;lower right\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003egrid\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eTrue\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elinestyle\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;--\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ealpha\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0.7\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eplt\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esavefig\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;jev_calibration_plot.png\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eplt\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eshow\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cfigure\u003e\n  \u003cimg src=\"../../images/jev_calibration_plot.png\" alt=\"jev_calibration_plot\"\u003e\n\u003c/figure\u003e\n\u003ch3 id=\"6-expected-calibration-error\"\u003e6. Expected Calibration Error\u003c/h3\u003e\n\u003cp\u003eSame definition as the original notebook, with one fix: \u003ccode\u003enp.digitize\u003c/code\u003e puts a probability of exactly \u003ccode\u003e1.0\u003c/code\u003e in bin index 10, which the loop never visits, so those predictions were dropped from the average. Jev (like the OpenAI mapping) returns many exact \u003ccode\u003e0.0\u003c/code\u003e/\u003ccode\u003e1.0\u003c/code\u003e values, and the \u003ccode\u003e1.0\u003c/code\u003e ones are where overconfidence shows up, so they have to be counted.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eexpected_calibration_error\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprob_pred\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003en_bins\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e10\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprob_pred\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003easarray\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003easarray\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eprob_pred\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ebins\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elinspace\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003en_bins\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e+\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ebinids\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eclip\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edigitize\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eprob_pred\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ebins\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003en_bins\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e   \u003cspan style=\"color:#75715e\"\u003e# p == 1.0 goes in the last bin\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eece\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e0.0\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eb\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erange\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003en_bins\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003emask\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ebinids\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eb\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emask\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eany\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e():\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003eece\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e+=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eabs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emask\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emean\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprob_pred\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emask\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emean\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e())\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e*\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emask\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esum\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e/\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elen\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eece\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eenumerate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esimplified_names\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eece\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eexpected_calibration_error\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e((\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eastype\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[:,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;ECE for \u0026#39;\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eece\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e*\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e100\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e:\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e.2f\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e%\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eECE for \u0026#39;mac\u0026#39;: 5.34%\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eECE for \u0026#39;motor\u0026#39;: 4.50%\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eECE for \u0026#39;baseball\u0026#39;: 1.96%\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eHelper used by the tables below. \u003cstrong\u003eTop-label ECE\u003c/strong\u003e asks whether the probability given to the predicted class matches how often that prediction is right (with a bootstrap 95% interval). Per-class ECE alone looks good when most probabilities are near 0, and ECE is biased upward at a few hundred samples, so read the interval before calling a difference real. \u003ccode\u003eacc p\u0026gt;=0.99\u003c/code\u003e is the direct test of \u0026ldquo;a probability of 1.0 should be right every time\u0026rdquo;.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eN_CLASSES\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elen\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esimplified_names\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003esummarize\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ey\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003en_boot\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1000\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eseed\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ey_pred\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etop_prob\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eargmax\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eaxis\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emax\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eaxis\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ecorrect\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_pred\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ey\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eastype\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003erng\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003erandom\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edefault_rng\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eseed\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eboots\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eexpected_calibration_error\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecorrect\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eidx\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etop_prob\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eidx\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e             \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eidx\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003erng\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eintegers\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elen\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elen\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e_\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erange\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003en_boot\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003elo\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ehi\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003epercentile\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eboots\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e2.5\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e97.5\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e*\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e100\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003esure\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etop_prob\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e\u0026gt;=\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e0.99\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003erow\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;model\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;n\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elen\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;accuracy\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecorrect\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emean\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;macro F1\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emetrics\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ef1_score\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ey_pred\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eaverage\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;macro\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;mean top prob\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etop_prob\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emean\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;top-label ECE %\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eexpected_calibration_error\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecorrect\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etop_prob\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e*\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e100\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;95% CI\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elo\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e:\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e.1f\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e, \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ehi\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e:\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e.1f\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e]\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Brier\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emean\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esum\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e((\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eeye\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eshape\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e**\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e2\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eaxis\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;n p\u0026gt;=0.99\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esure\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esum\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;acc p\u0026gt;=0.99\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecorrect\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esure\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emean\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esure\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eany\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eelse\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003enan\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ec\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eenumerate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esimplified_names\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003erow\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;ECE \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ec\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e %\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eexpected_calibration_error\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e((\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eastype\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[:,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e*\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e100\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erow\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"7-platt-recalibration\"\u003e7. Platt recalibration\u003c/h3\u003e\n\u003cp\u003ePlatt scaling fits one sigmoid per class on the logit of Jev\u0026rsquo;s probability (2 parameters per class), then renormalises the rows to sum to 1.\u003c/p\u003e\n\u003cp\u003eIt is scored \u003cstrong\u003eout-of-fold\u003c/strong\u003e: with 5-fold cross-validation each article is recalibrated by a model that never saw it, so all ~500 articles count and nothing is scored on the data it was fitted on. Probabilities of exactly 0 or 1 are clipped to \u003ccode\u003e1e-6\u003c/code\u003e first, because the logit of 0 or 1 is infinite.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eEPS\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1e-6\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003e_logit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ep\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ep\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eclip\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ep\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eEPS\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eEPS\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elog\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ep\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e/\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ep\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003efit_platt\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP_fit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ey_fit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eLogisticRegression\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eC\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1e3\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e_logit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP_fit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[:,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ek\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]]),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_fit\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ek\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eastype\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ek\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erange\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP_fit\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eshape\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eapply_platt\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emodels\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eP_new\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ecal\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecolumn_stack\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e([\u003c/span\u003e\u003cspan style=\"color:#111\"\u003em\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003epredict_proba\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e_logit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP_new\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[:,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ek\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]]))[:,\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ek\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003em\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eenumerate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emodels\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)])\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003etotals\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecal\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esum\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eaxis\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ekeepdims\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eTrue\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ewhere\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etotals\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecal\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e/\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ewhere\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etotals\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etotals\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e/\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecal\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eshape\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eout_of_fold_platt\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ey\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003en_splits\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e5\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eseed\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e42\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eout\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ezeros_like\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efit_idx\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etest_idx\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eStratifiedKFold\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003en_splits\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eshuffle\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eTrue\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erandom_state\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eseed\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esplit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ey\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003eout\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etest_idx\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eapply_platt\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efit_platt\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efit_idx\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ey\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efit_idx\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etest_idx\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eout\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eP_platt\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eout_of_fold_platt\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003efig\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eaxes\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eplt\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esubplots\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eN_CLASSES\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efigsize\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e18\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e5\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esharey\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eTrue\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eenumerate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ezip\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esimplified_names\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eaxes\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eplot\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e([\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;k:\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elabel\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Perfectly calibrated\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprobs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estyle\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elabel\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;o--\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Jev (raw)\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP_platt\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;s-\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Jev + Platt (out-of-fold)\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)]:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003efrac\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emean_p\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecalibration_curve\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprobs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[:,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003en_bins\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e10\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eplot\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emean_p\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efrac\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estyle\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elabel\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elabel\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ealpha\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0.8\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eset_title\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39; (N=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elen\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e)\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eset_xlabel\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Mean Predicted Probability\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003egrid\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eTrue\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elinestyle\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;--\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ealpha\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0.7\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eaxes\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eset_ylabel\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Fraction of Positives\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eaxes\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elegend\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eloc\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;upper left\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eplt\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etight_layout\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eplt\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esavefig\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;jev_platt_calibration_plot.png\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eplt\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eshow\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Accuracy: raw \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eargmax\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emean\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e:\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e.4f\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e | Platt \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP_platt\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eargmax\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emean\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e:\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e.4f\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eenumerate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esimplified_names\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ebefore\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eexpected_calibration_error\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e((\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eastype\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[:,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e*\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e100\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eafter\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eexpected_calibration_error\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e((\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eastype\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eP_platt\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[:,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e*\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e100\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;ECE \u0026#39;\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ebefore\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e:\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e.2f\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e% -\u0026gt; \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eafter\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e:\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e.2f\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e% after Platt\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eAccuracy: raw 0.9180 | Platt 0.9260\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eECE \u0026#39;mac\u0026#39;: 5.34% -\u0026gt; 2.23% after Platt\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eECE \u0026#39;motor\u0026#39;: 4.50% -\u0026gt; 2.10% after Platt\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eECE \u0026#39;baseball\u0026#39;: 1.96% -\u0026gt; 1.87% after Platt\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cfigure\u003e\n  \u003cimg src=\"../../images/jev_platt_calibration_plot.png\" alt=\"jev_platt_calibration_plot\"\u003e\n\u003c/figure\u003e\n\u003ch3 id=\"8-comparison-with-the-openai-run\"\u003e8. Comparison with the OpenAI run\u003c/h3\u003e\n\u003cp\u003e\u003ccode\u003eopenai_classification_results.csv\u003c/code\u003e does not keep the article index (and has 496 valid rows), so the OpenAI run is a different sample of the same distribution: compare it as a reference rather than row by row. Its probabilities come from the first-token logprob heuristic in \u003ccode\u003econfidence.ipynb\u003c/code\u003e, renormalised over the matched tokens; Jev\u0026rsquo;s are its native \u003ccode\u003eprobabilities\u003c/code\u003e.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eruns\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Jev\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Jev + Platt (out-of-fold)\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey_true\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eP_platt\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eos\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003epath\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eexists\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;openai_classification_results.csv\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eopenai_df\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003epd\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eread_csv\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;openai_classification_results.csv\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eruns\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;OpenAI gpt-4o-mini\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eopenai_df\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;y_true_final\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eto_numpy\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eopenai_df\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eprob_cols\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eto_numpy\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003eelse\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;openai_classification_results.csv not found; showing Jev only.\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003esummary\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003epd\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eDataFrame\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e([\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esummarize\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ey\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprobs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprobs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eruns\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eitems\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()])\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eset_index\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;model\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003edisplay\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esummary\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eround\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e3\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003emetric\u003c/th\u003e\n          \u003cth\u003eJev\u003c/th\u003e\n          \u003cth\u003eJev + Platt (out-of-fold)\u003c/th\u003e\n          \u003cth\u003eOpenAI gpt-4o-mini\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003en\u003c/td\u003e\n          \u003ctd\u003e500\u003c/td\u003e\n          \u003ctd\u003e500\u003c/td\u003e\n          \u003ctd\u003e496\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eaccuracy\u003c/td\u003e\n          \u003ctd\u003e0.918\u003c/td\u003e\n          \u003ctd\u003e0.926\u003c/td\u003e\n          \u003ctd\u003e0.911\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003emacro F1\u003c/td\u003e\n          \u003ctd\u003e0.919\u003c/td\u003e\n          \u003ctd\u003e0.926\u003c/td\u003e\n          \u003ctd\u003e0.913\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003emean top prob\u003c/td\u003e\n          \u003ctd\u003e0.942\u003c/td\u003e\n          \u003ctd\u003e0.920\u003c/td\u003e\n          \u003ctd\u003e0.986\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003etop-label ECE %\u003c/td\u003e\n          \u003ctd\u003e3.524\u003c/td\u003e\n          \u003ctd\u003e1.248\u003c/td\u003e\n          \u003ctd\u003e7.454\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e95% CI\u003c/td\u003e\n          \u003ctd\u003e[2.5, 5.8]\u003c/td\u003e\n          \u003ctd\u003e[1.1, 3.4]\u003c/td\u003e\n          \u003ctd\u003e[5.4, 9.8]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eBrier\u003c/td\u003e\n          \u003ctd\u003e0.115\u003c/td\u003e\n          \u003ctd\u003e0.099\u003c/td\u003e\n          \u003ctd\u003e0.154\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003en p\u0026gt;=0.99\u003c/td\u003e\n          \u003ctd\u003e356\u003c/td\u003e\n          \u003ctd\u003e328\u003c/td\u003e\n          \u003ctd\u003e451\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eacc p\u0026gt;=0.99\u003c/td\u003e\n          \u003ctd\u003e0.997\u003c/td\u003e\n          \u003ctd\u003e0.997\u003c/td\u003e\n          \u003ctd\u003e0.967\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eECE mac %\u003c/td\u003e\n          \u003ctd\u003e5.338\u003c/td\u003e\n          \u003ctd\u003e2.232\u003c/td\u003e\n          \u003ctd\u003e6.813\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eECE motor %\u003c/td\u003e\n          \u003ctd\u003e4.498\u003c/td\u003e\n          \u003ctd\u003e2.102\u003c/td\u003e\n          \u003ctd\u003e5.224\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eECE baseball %\u003c/td\u003e\n          \u003ctd\u003e1.962\u003c/td\u003e\n          \u003ctd\u003e1.868\u003c/td\u003e\n          \u003ctd\u003e3.437\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003ecolors\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Jev\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;tab:blue\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Jev + Platt (out-of-fold)\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;tab:green\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;OpenAI gpt-4o-mini\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;tab:orange\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003efig\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eaxes\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eplt\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esubplots\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eN_CLASSES\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e+\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efigsize\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e22\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e5\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esharey\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eTrue\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etitle\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ezip\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eaxes\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e*\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esimplified_names\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;top label (all classes)\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eplot\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e([\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;k:\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elabel\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Perfectly calibrated\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eset_title\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etitle\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eset_xlabel\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Mean Predicted Probability\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eax\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003egrid\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eTrue\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elinestyle\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;--\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ealpha\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0.7\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprobs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eruns\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eitems\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e():\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erange\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eN_CLASSES\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003efrac\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emean_p\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecalibration_curve\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ey\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprobs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[:,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003en_bins\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e10\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003eaxes\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eplot\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emean_p\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efrac\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;o-\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecolor\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecolors\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elabel\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003efrac\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emean_p\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecalibration_curve\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eprobs\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eargmax\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ey\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprobs\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emax\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003en_bins\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e10\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eaxes\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e-\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eplot\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emean_p\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efrac\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;o-\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecolor\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecolors\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elabel\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eaxes\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eset_ylabel\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Fraction of Positives\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eaxes\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elegend\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eloc\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;upper left\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eplt\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etight_layout\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eplt\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esavefig\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;jev_vs_openai_calibration.png\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eplt\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eshow\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cfigure\u003e\n  \u003cimg src=\"../../images/jev_vs_openai_calibration.png\" alt=\"jev_vs_openai_calibration\"\u003e\n\u003c/figure\u003e\n","url":"http://gabelfra1.github.io/gabelfra.github.io/posts/jev/","date_published":"21096-21-09T90:2121:00+00:00","date_modified":"21096-21-09T90:2121:00+00:00","author":{"name":"Calvin Tran","url":"http://gabelfra1.github.io/gabelfra.github.io/"}},{"id":"40ce30011f806e91eb865d2fb485ea6fd6d55930","title":"Classification is back","summary":"","content_text":"The decision was always the product A while ago I wrote about how badly calibrated LLMs are. Any GPT-model beat a traditional ML model (TF-IDF \u0026amp; Naive Bayes) at classifcation but it\u0026rsquo;s probability scores are just too uncalibrated to be useful.\nThen I wrote a code along on agentic patterns, and every one of those patterns has the same problem underneath. Routing is a classifier, the evaluator is a classifier, the tool selector is a classifier and so on..\nPut the two articles together and you have the real problem with modern automation. What we build is workflows. What a workflow needs at every node is a decision with an honest probability attached. What the model is optimised to produce is text not calibrated decisions.\nWhat automation actually asks for Strip the marketing off any enterprise use case and the loop is the same. Ingest some context, most of it unstructured. Make a decision. Act on it with a tool. Repeat.\nNone of those steps needs the capability to produce text. Nobody in production reads the model\u0026rsquo;s reasoning trace of why it routed the ticket to the credit card team. The explanation exists because text is the only interface on offer, so we generate it and throw it away after parsing the output to generate a structured response.\nThe parsing is the tell. If your first action on receiving a model output is to run a validator over it, the model is not speaking your language.\nThe hoops Everyone building these systems carries the same scaffolding. A Pydantic schema plus some additional tricks , because the model can emit a category that does not exist. Temperature zero, which does not make the system deterministic, it just makes it feel deterministic. Retry loops, so you make five calls instead of one.\nThis is not a prompting problem. Guo et al. (2017) showed that modern networks drift towards overconfidence as a property of how we train them, and RLHF pushes the same way by optimising for what a human rater prefers to read. A model rewarded for sounding helpful will sound certain.\nSystem One models TypeSafe released Jev on 15 September, the first of what they call System One models. The name comes from Kahneman\u0026rsquo;s split between fast intuitive judgement and slow deliberate reasoning.\nThe design decision is the interesting part. Jev gives up text generation entirely. It samples all of its outputs in parallel in a single query rather than one token at a time, and it is trained with a method they call Reinforcement Learning for Calibrated Decisions instead of RLHF. You send a state and a set of typed questions, and you get back typed values and probability distributions.\nEverything in the previous section becomes unnecessary if that holds. There is no parsing because there is no string. There are no type errors, not because they are rare but because the possible outputs are defined in advance and schema matching is guaranteed. And the probability is not fugazi, it is what the model was trained to produce.\nCalibration is king Calibration means that among all the predictions the model assigns 70% to, roughly 70% turn out correct. Without that property you cannot set a threshold, and without a threshold every decision needs a human in the loop.\nJav returns the full distribution and also collapses its shape into a single confidence number, so you can threshold without computing it yourself. A flat distribution means low confidence: no option is a clear winner, or the state does not contain enough to go on. What follows is the pattern every risk function already understands. A floor below which you route to a human, and above it a bar for acting without confirmation that scales with what the action costs if it is wrong.\nThe part I am not sold on yet The workflow evals are the weak point, agreement with a panel of frontier LLMs is not the same as being right (try LLM as a judge), and it is a strange yardstick for a model whose pitch is that frontier LLMs are badly calibrated. The one claim genuinely free of this problem is type safety, which is guaranteed by construction and falsifiable with a single API call.\nAnd the obvious one. This does not replace LLMs. It replaces the part of your agent that was pretending to be a classifier. When you need generated language,images or video you still need a generative model. What changes is that the decision layer stops being made of the same material as the generation layer.\nWhat I want to see, and intend to run myself (if I get my hand on the early access), is a straight calibration benchmark. Same 20 Newsgroups setup as the first article, reliability diagram, ECE, Brier score, no vendor in the loop.\nConclusion I ended the calibration article by saying that where a decision depends on a trustworthy measure of confidence, modern LLMs are a bad choice, and you should accept the cost of labels and training something simpler. I still think that was right for the world as it was.\nBut if the generality (of inputs) and the calibration (of outputs) hold up across common domains, RLCD could turn out to be the missing piece in every automation workflow.\nAnd I would be very happy to delete the scaffolding we built around LLMs to force them into being type safe and marginally less overconfident, and to pay less for the privilege.\n","content_html":"\u003ch3 id=\"the-decision-was-always-the-product\"\u003eThe decision was always the product\u003c/h3\u003e\n\u003cp\u003eA while ago I wrote about \u003ca href=\"/blog/llm-are-way-too-confident\"\u003ehow badly calibrated LLMs are\u003c/a\u003e. Any GPT-model beat a traditional ML model (TF-IDF \u0026amp; Naive Bayes) at classifcation but it\u0026rsquo;s probability scores are just too uncalibrated to be useful.\u003c/p\u003e\n\u003cp\u003eThen I wrote a \u003ca href=\"/blog/building-effective-agents\"\u003ecode along on agentic patterns\u003c/a\u003e, and every one of those patterns has the same problem underneath. Routing is a classifier, the evaluator is a classifier, the tool selector is a classifier and so on..\u003c/p\u003e\n\u003cp\u003ePut the two articles together and you have the real problem with modern automation. What we build is workflows. What a workflow needs at every node is a decision with an honest probability attached. What the model is optimised to produce is text not calibrated decisions.\u003c/p\u003e\n\u003ch4 id=\"what-automation-actually-asks-for\"\u003eWhat automation actually asks for\u003c/h4\u003e\n\u003cp\u003eStrip the marketing off any enterprise use case and the loop is the same. Ingest some context, most of it unstructured. Make a decision. Act on it with a tool. Repeat.\u003c/p\u003e\n\u003cp\u003eNone of those steps needs the capability to produce text. Nobody in production reads the model\u0026rsquo;s reasoning trace of why it routed the ticket to the credit card team. The explanation exists because text is the only interface on offer, so we generate it and throw it away after parsing the output to generate a structured response.\u003c/p\u003e\n\u003cp\u003eThe parsing is the tell. If your first action on receiving a model output is to run a validator over it, the model is not speaking your language.\u003c/p\u003e\n\u003ch4 id=\"the-hoops\"\u003eThe hoops\u003c/h4\u003e\n\u003cp\u003eEveryone building these systems carries the same scaffolding. A Pydantic schema plus some additional tricks , because the model can emit a category that does not exist. Temperature zero, which does not make the system deterministic, it just makes it feel deterministic.\nRetry loops, so you make five calls instead of one.\u003c/p\u003e\n\u003cp\u003eThis is not a prompting problem. \u003ca href=\"https://arxiv.org/abs/1706.04599\"\u003eGuo et al. (2017)\u003c/a\u003e showed that modern networks drift towards overconfidence as a property of how we train them, and RLHF pushes the same way by optimising for what a human rater prefers to read. A model rewarded for sounding helpful will sound certain.\u003c/p\u003e\n\u003ch4 id=\"system-one-models\"\u003eSystem One models\u003c/h4\u003e\n\u003cp\u003eTypeSafe released \u003ca href=\"https://typesafe.ai/blog/introducing-system-one-models-and-jev\"\u003eJev\u003c/a\u003e on 15 September, the first of what they call \u003cstrong\u003eSystem One models\u003c/strong\u003e. The name comes from Kahneman\u0026rsquo;s split between fast intuitive judgement and slow deliberate reasoning.\u003c/p\u003e\n\u003cp\u003eThe design decision is the interesting part. Jev gives up text generation entirely. It samples all of its outputs in parallel in a single query rather than one token at a time, and it is trained with a method they call Reinforcement Learning for Calibrated Decisions instead of RLHF. You send a state and a set of typed questions, and you get back typed values and probability distributions.\u003c/p\u003e\n\u003cp\u003eEverything in the previous section becomes unnecessary if that holds. There is no parsing because there is no string. There are no type errors, not because they are rare but because the possible outputs are defined in advance and schema matching is guaranteed. And the probability is not fugazi, it is what the model was trained to produce.\u003c/p\u003e\n\u003ch4 id=\"calibration-is-king\"\u003eCalibration is king\u003c/h4\u003e\n\u003cp\u003e\u003cstrong\u003eCalibration means that among all the predictions the model assigns 70% to, roughly 70% turn out correct.\u003c/strong\u003e Without that property you cannot set a threshold, and without a threshold every decision needs a human in the loop.\u003c/p\u003e\n\u003cp\u003eJav returns the full distribution and also collapses its shape into a single confidence number, so you can threshold without computing it yourself. A flat distribution means low confidence: no option is a clear winner, or the state does not contain enough to go on. What follows is the pattern every risk function already understands.\nA floor below which you route to a human, and above it a bar for acting without confirmation that scales with what the action costs if it is wrong.\u003c/p\u003e\n\u003ch4 id=\"the-part-i-am-not-sold-on-yet\"\u003eThe part I am not sold on yet\u003c/h4\u003e\n\u003cp\u003eThe workflow evals are the weak point, agreement with a panel of frontier LLMs is not the same as being right (try LLM as a judge), and it is a strange yardstick for a model whose pitch is that frontier LLMs are badly calibrated. The one claim genuinely free of this problem is type safety, which is guaranteed by construction and falsifiable with a single API call.\u003c/p\u003e\n\u003cp\u003eAnd the obvious one. This does not replace LLMs. It replaces the part of your agent that was pretending to be a classifier. When you need generated language,images or video you still need a generative model. What changes is that the decision layer stops being made of the same material as the generation layer.\u003c/p\u003e\n\u003cp\u003eWhat I want to see, and intend to run myself (if I get my hand on the early access), is a straight calibration benchmark. Same 20 Newsgroups setup as the first article, reliability diagram, ECE, Brier score, no vendor in the loop.\u003c/p\u003e\n\u003ch3 id=\"conclusion\"\u003eConclusion\u003c/h3\u003e\n\u003cp\u003eI ended the calibration article by saying that where a decision depends on a trustworthy measure of confidence, modern LLMs are a bad choice, and you should accept the cost of labels and training something simpler. I still think that was right for the world as it was.\u003c/p\u003e\n\u003cp\u003eBut if the generality (of inputs) and the calibration (of outputs) hold up across common domains, RLCD could turn out to be the missing piece in every automation workflow.\u003c/p\u003e\n\u003cp\u003eAnd I would be very happy to delete the scaffolding we built around LLMs to force them into being type safe and marginally less overconfident, and to pay less for the privilege.\u003c/p\u003e\n","url":"http://gabelfra1.github.io/gabelfra.github.io/posts/classification/","date_published":"17096-17-09T90:1717:00+00:00","date_modified":"17096-17-09T90:1717:00+00:00","author":{"name":"Calvin Tran","url":"http://gabelfra1.github.io/gabelfra.github.io/"}},{"id":"61b971f971f86335f79e01a467adcb5e63aca01a","title":"Who Watches the Watchmen?","summary":"","content_text":"This article is about single-turn LLM generation: producing text, answering questions, summarising, and so on. I am not going to cover multi-turn conversation or agentic systems. The question is a practical one. You have shipped, or are about to ship, a system that relies on an LLM for real users in a real business. How do you know if it is any good?\nThe old world: NLP measures Before large language models entered the picture, NLP evaluation had a clean, mathematical answer: measure how similar the output is to a known good reference. The dominant metric was BLEU (Papineni et al., 2002), the Bilingual Evaluation Understudy, imported from the machine translation research that would later give us the Transformer (Vaswani et al., 2017). BLEU counts overlapping n-grams between a generated string and a reference string and produces a score between 0 and 1. ROUGE (Lin, 2004) did the same for summarisation. Cosine similarity over TF-IDF or word embeddings offered a softer version of the same idea.\nThese metrics had real virtues. They were fast, cheap, deterministic, and needed no human in the loop once you had a reference corpus.\nThe problem is that language is not a bag of words, and meaning is not proximity in token space. \u0026ldquo;The patient is not responding to treatment\u0026rdquo; and \u0026ldquo;The patient is responding well to treatment\u0026rdquo; share most of their surface and would score highly against one another. \u0026ldquo;The medication was administered\u0026rdquo; and \u0026ldquo;The medication was not administered\u0026rdquo; sit a single negation apart. Similarity-based metrics are structurally blind to these distinctions, and in a real world scenario that blindness becomes fatal.\nClosing the gap: Natural Language Inference The field\u0026rsquo;s next attempt to do better came through Natural Language Inference (NLI): the task of deciding whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise. Models fine-tuned on datasets like SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018) learned something closer to semantic reasoning than to surface matching.\nThe training data looks like this:\nPremise Hypothesis Label A football game with kids playing. Some kids are playing. Entailment A man is in a kitchen cooking The man is sleeping. Contradiction The idea, applied to evaluation, was to check not whether the output looked like the reference, but whether it was logically coherent with, it was a genuine improvement. NLI-based scoring caught contradictions that BLEU missed; it was sensitive to negation, to entailment direction, to whether a claim was supported or merely adjacent to the reference.\nBut NLI brought its own problem, and you can already see it in the table above. Those premises and hypotheses are short, tidy, single-clause sentences derived from image captions. Real documents do not read like the Elements of Euclid. Real world documents are messy, contextual, sentences long, full of hedges and asides and clauses that depend on one another. An entailment model trained on caption pairs has very little to say about whether a four-paragraph answer faithfully reflects a retrieved policy document. It was a well-designed experiment that, for the most part, stayed in the lab.\nThe new world : LLM-as-Judge Once capable LLMs existed, the obvious move was to use them to evaluate each other. Feed the system prompt, the user query and the model\u0026rsquo;s response to a capable judge model and get back a score. This approach, LLM-as-Judge, was studied systematically by Zheng et al. (2023) and has since become the de facto standard. A capable judge can assess fluency, faithfulness, relevance and tone in a single pass, and it can explain its reasoning on edge cases that would confuse any fixed metric.\nA growing ecosystem of packages wraps this pattern into something you can drop into a pipeline. RAGAS (Es et al., 2024) is one example among many: built for retrieval-augmented generation, it scores dimensions like faithfulness (does the answer stick to the retrieved context?), answer relevancy and context recall, each judged independently so you get a full picture rather than one opaque number. DeepEval, TruLens and others occupy the same space with different trade-offs. The framework matters less than understanding what it is doing underneath, which is asking an LLM to grade another LLM.\nThat is where the real problem lives, and it is one of epistemology: you are using a model with its own biases to evaluate a model with its own biases. Three failure modes are well documented. Self-preference bias: judges favour outputs that resemble their own, and Panickssery et al. (2024) link this directly to a model\u0026rsquo;s ability to recognise its own generations, so using GPT-4 to judge GPT-4 is not a neutral act. Verbosity bias: longer, more hedged answers score higher than concise, correct ones.Position bias: in a pairwise comparison the answer shown first wins more often.\nMitigations exist: prompt the judge to reason before scoring, run each comparison in both orders and average, use a different model family as judge than the one under test. None of them fully closes the gap. LLM-as-Judge is powerful tool but has it\u0026rsquo;s cracks.\nBack to basics: the golden dataset The canonical machine learning answer is the test dataset: a golden dataset of inputs paired with verified outputs, against which any version of the system can be scored and compared. The appeal is obvious. It is deterministic, reproducible and immune to judge bias.\nThe problem is circular in an uncomfortable way. To score an LLM\u0026rsquo;s free-form generation against a golden answer, you need a method to compare the produced response to the expected one, and you are right back where you started. A similarity metric misses semantic divergence; an NLI model carries the problems described above; a human reviewing every pair does not scale. You can use an LLM judge for the comparison, but then your golden evaluation is only as good as that judge, and you have not escaped the watchmen problem, you have just moved it up one layer.\nGolden datasets still earn their place when you narrow what they test. For constrained tasks (extracting a specific field, classifying into a fixed set of categories, requiring a response to contain certain factual claims) binary or near-binary correctness checks are both tractable and meaningful. The failure mode is stretching this to open-ended generation, where the space of acceptable answers is large and your reference is only one valid output among many.\nWhen you do need to measure closeness to an expected answer, the most reliable options today are embedding-based similarity using a strong general-purpose embedding model, or structured decomposition: break the expected answer into individual factual claims and verify each one separately rather than comparing the whole.\nThe user as ground truth There is one more signal, and it sits in production: the user. Thumbs up and thumbs down, copy-to-clipboard events, a follow-up question that reveals the first answer was wrong, session usage. These are all expressions of satisfaction or dissatisfaction that need no instrumentation beyond basic analytics.\nThe seductive thing about implicit feedback is that it is free, continuous and available at scale. The dangerous thing is that it is confounded by almost everything: UI, task difficulty, user expertise, stylistic preferences that have nothing to do with quality. A thumbs down might mean \u0026ldquo;the answer was wrong\u0026rdquo;, or \u0026ldquo;the answer was right but too boring\u0026rdquo;, or \u0026ldquo;I clicked it by accident\u0026rdquo;, or \u0026ldquo;I was already frustrated before I opened the tool\u0026rdquo;. Explicit Likert ratings remove some of that ambiguity but add survey fatigue and selection bias, since the users who rate are rarely representative of the users who do not.\nTreat user feedback as a monitoring signal, not a validation metric. It is good for catching regressions, surfacing categories of failure and deciding what to investigate next. It is not reliable enough to be the primary basis for an iteration decision.\nA practical framework Putting it all together, here is a reasonable approach for a team shipping a production LLM system, at least in my opinion\nDefine what correctness means before you build anything. The evaluation strategy should fall out of the task definition, not get bolted on afterwards. If you cannot write down in concrete terms what a good response looks like, you are not ready.\nUse automated metrics as filters, not as truth. LLM-as-Judge metrics catch obvious failures reliably and subtle ones unreliably. Use them to shrink the volume of output that needs human eyes, not to replace those eyes.\nBuild a golden dataset for regression testing A modest, carefully curated set of inputs with verified references beats a large noisy one.That set is your foundation for changing the system without breaking it in production.\nShow the retrieved context or the reasoning trace to the user. If the job is to convey accurate information, expose the sources behind each answer so the user can confirm or reject them. This turns your users into a distributed check on your failure modes.\nTreat user feedback as continuous monitoring. Log it, track it over time, investigate anomalies, and keep iterating.\nEvaluate offline first, then ship with monitoring. The ship of certainty sailed long ago. The goal is not to be perfect before deployment, which is unattainable, but to have enough signal to catch a mistake quickly and roll back.\nStay sceptical of every metric All of this is probabilistic. No single number tells you the system is good; you are watching whether the whole picture is trending in the right direction. Any metric can be gamed, can drift, or can look healthy while something underneath rots. Trust the convergence of several weak signals over any one strong-looking score.\nConclusion There is no clean solution to this. Every evaluation method has a crack, and every crack is filled by some other method you are trusting implicitly. What you can do is be honest about where each layer breaks down, stack layers that fail in different ways, and keep humans in contact with real output often enough to catch what the metrics miss. That is not a perfect answer. It is the only one on offer.\nReferences Papineni, K., Roukos, S., Ward, T., Zhu, W. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation. ACL. Lin, C. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out, ACL Workshop. Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS. Bowman, S., Angeli, G., Potts, C., Manning, C. (2015). A Large Annotated Corpus for Learning Natural Language Inference. EMNLP. Williams, A., Nangia, N., Bowman, S. (2018). A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. NAACL. Zhang, T., Kishore, V., Wu, F., Weinberger, K., Artzi, Y. (2020). BERTScore: Evaluating Text Generation with BERT. ICLR. Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS. Es, S., James, J., Espinosa-Anke, L., Schockaert, S. (2024). RAGAS: Automated Evaluation of Retrieval Augmented Generation. EACL. Panickssery, A., Bowman, S., Feng, S. (2024). LLM Evaluators Recognize and Favor Their Own Generations. NeurIPS. ","content_html":"\u003cp\u003e\u003cem\u003eThis article is about single-turn LLM generation: producing text, answering questions, summarising, and so on. I am not going to cover multi-turn conversation or agentic systems. The question is a practical one. You have shipped, or are about to ship, a system that relies on an LLM for real users in a real business. How do you know if it is any good?\u003c/em\u003e\u003c/p\u003e\n\u003ch3 id=\"the-old-world-nlp-measures\"\u003eThe old world: NLP measures\u003c/h3\u003e\n\u003cp\u003eBefore large language models entered the picture, NLP evaluation had a clean, mathematical answer: measure how similar the output is to a known good reference. The dominant metric was BLEU (\u003ca href=\"https://aclanthology.org/P02-1040/\"\u003ePapineni et al., 2002\u003c/a\u003e), the Bilingual Evaluation Understudy, imported from the machine translation research that would later give us the Transformer (\u003ca href=\"https://arxiv.org/abs/1706.03762\"\u003eVaswani et al., 2017\u003c/a\u003e). BLEU counts overlapping n-grams between a generated string and a reference string and produces a score between 0 and 1. ROUGE (\u003ca href=\"https://aclanthology.org/W04-1013/\"\u003eLin, 2004\u003c/a\u003e) did the same for summarisation. Cosine similarity over TF-IDF or word embeddings offered a softer version of the same idea.\u003c/p\u003e\n\u003cp\u003eThese metrics had real virtues. They were fast, cheap, deterministic, and needed no human in the loop once you had a reference corpus.\u003c/p\u003e\n\u003cp\u003eThe problem is that language is not a bag of words, and meaning is not proximity in token space. \u0026ldquo;The patient is not responding to treatment\u0026rdquo; and \u0026ldquo;The patient is responding well to treatment\u0026rdquo; share most of their surface and would score highly against one another. \u0026ldquo;The medication was administered\u0026rdquo; and \u0026ldquo;The medication was not administered\u0026rdquo; sit a single negation apart. Similarity-based metrics are structurally blind to these distinctions, and in a real world scenario that blindness becomes fatal.\u003c/p\u003e\n\u003ch3 id=\"closing-the-gap-natural-language-inference\"\u003eClosing the gap: Natural Language Inference\u003c/h3\u003e\n\u003cp\u003eThe field\u0026rsquo;s next attempt to do better came through Natural Language Inference (NLI): the task of deciding whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise. Models fine-tuned on datasets like SNLI (\u003ca href=\"https://arxiv.org/abs/1508.05326\"\u003eBowman et al., 2015\u003c/a\u003e) and MNLI (\u003ca href=\"https://arxiv.org/abs/1704.05426\"\u003eWilliams et al., 2018\u003c/a\u003e) learned something closer to semantic reasoning than to surface matching.\u003c/p\u003e\n\u003cp\u003eThe training data looks like this:\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003ePremise\u003c/th\u003e\n          \u003cth\u003eHypothesis\u003c/th\u003e\n          \u003cth\u003eLabel\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eA football game with kids playing.\u003c/td\u003e\n          \u003ctd\u003eSome kids are playing.\u003c/td\u003e\n          \u003ctd\u003eEntailment\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eA man is in a kitchen cooking\u003c/td\u003e\n          \u003ctd\u003eThe man is sleeping.\u003c/td\u003e\n          \u003ctd\u003eContradiction\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eThe idea, applied to evaluation, was to check not whether the output looked like the reference, but whether it was logically coherent with, it was a genuine improvement. NLI-based scoring caught contradictions that BLEU missed; it was sensitive to negation, to entailment direction, to whether a claim was supported or merely adjacent to the reference.\u003c/p\u003e\n\u003cp\u003eBut NLI brought its own problem, and you can already see it in the table above. Those premises and hypotheses are short, tidy, single-clause sentences derived from image captions. Real documents do not read like the Elements of Euclid. Real world documents are messy, contextual,  sentences long, full of hedges and asides and clauses that depend on one another. An entailment model trained on caption pairs has very little to say about whether a four-paragraph answer faithfully reflects a retrieved policy document. It was a well-designed experiment that, for the most part, stayed in the lab.\u003c/p\u003e\n\u003ch3 id=\"the-new-world--llm-as-judge\"\u003eThe new world : LLM-as-Judge\u003c/h3\u003e\n\u003cp\u003eOnce capable LLMs existed, the obvious move was to use them to evaluate each other. Feed the system prompt, the user query and the model\u0026rsquo;s response to a capable judge model and get back a score. This approach, LLM-as-Judge, was studied systematically by \u003ca href=\"https://arxiv.org/abs/2306.05685\"\u003eZheng et al. (2023)\u003c/a\u003e and has since become the de facto standard. A capable judge can assess fluency, faithfulness, relevance and tone in a single pass, and it can explain its reasoning on edge cases that would confuse any fixed metric.\u003c/p\u003e\n\u003cp\u003eA growing ecosystem of packages wraps this pattern into something you can drop into a pipeline. RAGAS (\u003ca href=\"https://arxiv.org/abs/2309.15217\"\u003eEs et al., 2024\u003c/a\u003e) is one example among many: built for retrieval-augmented generation, it scores dimensions like faithfulness (does the answer stick to the retrieved context?), answer relevancy and context recall, each judged independently so you get a full picture rather than one opaque number. DeepEval, TruLens and others occupy the same space with different trade-offs. \u003cstrong\u003eThe framework matters less than understanding what it is doing underneath, which is asking an LLM to grade another LLM.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThat is where the real problem lives, and it is one of epistemology: you are using a model with its own biases to evaluate a model with its own biases. Three failure modes are well documented. \u003cstrong\u003eSelf-preference bias\u003c/strong\u003e: judges favour outputs that resemble their own, and \u003ca href=\"https://arxiv.org/abs/2404.13076\"\u003ePanickssery et al. (2024)\u003c/a\u003e link this directly to a model\u0026rsquo;s ability to recognise its own generations, so using GPT-4 to judge GPT-4 is not a neutral act. \u003cstrong\u003eVerbosity bias\u003c/strong\u003e: longer, more hedged answers score higher than concise, correct ones.\u003cstrong\u003ePosition bias\u003c/strong\u003e: in a pairwise comparison the answer shown first wins more often.\u003c/p\u003e\n\u003cp\u003eMitigations exist: prompt the judge to reason before scoring, run each comparison in both orders and average, use a different model family as judge than the one under test. None of them fully closes the gap. LLM-as-Judge is powerful tool but has it\u0026rsquo;s cracks.\u003c/p\u003e\n\u003ch3 id=\"back-to-basics-the-golden-dataset\"\u003eBack to basics: the golden dataset\u003c/h3\u003e\n\u003cp\u003eThe canonical machine learning answer is the test dataset: a golden dataset of inputs paired with verified outputs, against which any version of the system can be scored and compared. The appeal is obvious. It is deterministic, reproducible and immune to judge bias.\u003c/p\u003e\n\u003cp\u003eThe problem is circular in an uncomfortable way. To score an LLM\u0026rsquo;s free-form generation against a golden answer, you need a method to compare the produced response to the expected one, and you are right back where you started. A similarity metric misses semantic divergence; an NLI model carries the problems described above; a human reviewing every pair does not scale. You can use an LLM judge for the comparison, but then your golden evaluation is only as good as that judge, and you have not escaped the watchmen problem, you have just moved it up one layer.\u003c/p\u003e\n\u003cp\u003eGolden datasets still earn their place when you narrow what they test. For constrained tasks (extracting a specific field, classifying into a fixed set of categories, requiring a response to contain certain factual claims) binary or near-binary correctness checks are both tractable and meaningful. The failure mode is stretching this to open-ended generation, where the space of acceptable answers is large and your reference is only one valid output among many.\u003c/p\u003e\n\u003cp\u003eWhen you do need to measure closeness to an expected answer, the most reliable options today are embedding-based similarity using a strong general-purpose embedding model, or structured decomposition: break the expected answer into individual factual claims and verify each one separately rather than comparing the whole.\u003c/p\u003e\n\u003ch3 id=\"the-user-as-ground-truth\"\u003eThe user as ground truth\u003c/h3\u003e\n\u003cp\u003eThere is one more signal, and it sits in production: the user. Thumbs up and thumbs down, copy-to-clipboard events, a follow-up question that reveals the first answer was wrong, session usage. These are all expressions of satisfaction or dissatisfaction that need no instrumentation beyond basic analytics.\u003c/p\u003e\n\u003cp\u003eThe seductive thing about implicit feedback is that it is free, continuous and available at scale. The dangerous thing is that it is confounded by almost everything: UI, task difficulty, user expertise, stylistic preferences that have nothing to do with quality. A thumbs down might mean \u0026ldquo;the answer was wrong\u0026rdquo;, or \u0026ldquo;the answer was right but too boring\u0026rdquo;, or \u0026ldquo;I clicked it by accident\u0026rdquo;, or \u0026ldquo;I was already frustrated before I opened the tool\u0026rdquo;. Explicit Likert ratings remove some of that ambiguity but add survey fatigue and selection bias, since the users who rate are rarely representative of the users who do not.\u003c/p\u003e\n\u003cp\u003eTreat user feedback as a monitoring signal, not a validation metric. It is good for catching regressions, surfacing categories of failure and deciding what to investigate next. It is not reliable enough to be the primary basis for an iteration decision.\u003c/p\u003e\n\u003ch3 id=\"a-practical-framework\"\u003eA practical framework\u003c/h3\u003e\n\u003cp\u003ePutting it all together, here is a reasonable approach for a team shipping a production LLM system, at least in my opinion\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDefine what correctness means before you build anything.\u003c/strong\u003e The evaluation strategy should fall out of the task definition, not get bolted on afterwards. If you cannot write down in concrete terms what a good response looks like, you are not ready.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUse automated metrics as filters, not as truth.\u003c/strong\u003e LLM-as-Judge metrics catch obvious failures reliably and subtle ones unreliably. Use them to shrink the volume of output that needs human eyes, not to replace those eyes.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eBuild a golden dataset for regression testing\u003c/strong\u003e A modest, carefully curated set of inputs with verified references beats a large noisy one.That set is your foundation for changing the system without breaking it in production.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eShow the retrieved context or the reasoning trace to the user.\u003c/strong\u003e If the job is to convey accurate information, expose the sources behind each answer so the user can confirm or reject them. This turns your users into a distributed check on your failure modes.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTreat user feedback as continuous monitoring.\u003c/strong\u003e Log it, track it over time, investigate anomalies, and keep iterating.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEvaluate offline first, then ship with monitoring.\u003c/strong\u003e The ship of certainty sailed long ago. The goal is not to be perfect before deployment, which is unattainable, but to have enough signal to catch a mistake quickly and roll back.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStay sceptical of every metric\u003c/strong\u003e All of this is probabilistic. No single number tells you the system is good; you are watching whether the whole picture is trending in the right direction. Any metric can be gamed, can drift, or can look healthy while something underneath rots. Trust the convergence of several weak signals over any one strong-looking score.\u003c/p\u003e\n\u003ch3 id=\"conclusion\"\u003eConclusion\u003c/h3\u003e\n\u003cp\u003eThere is no clean solution to this. Every evaluation method has a crack, and every crack is filled by some other method you are trusting implicitly. What you can do is be honest about where each layer breaks down, stack layers that fail in different ways, and keep humans in contact with real output often enough to catch what the metrics miss. That is not a perfect answer. It is the only one on offer.\u003c/p\u003e\n\u003ch4 id=\"references\"\u003eReferences\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003ePapineni, K., Roukos, S., Ward, T., Zhu, W. (2002). \u003ca href=\"https://aclanthology.org/P02-1040/\"\u003eBLEU: a Method for Automatic Evaluation of Machine Translation\u003c/a\u003e. ACL.\u003c/li\u003e\n\u003cli\u003eLin, C. (2004). \u003ca href=\"https://aclanthology.org/W04-1013/\"\u003eROUGE: A Package for Automatic Evaluation of Summaries\u003c/a\u003e. Text Summarization Branches Out, ACL Workshop.\u003c/li\u003e\n\u003cli\u003eVaswani, A., et al. (2017). \u003ca href=\"https://arxiv.org/abs/1706.03762\"\u003eAttention Is All You Need\u003c/a\u003e. NeurIPS.\u003c/li\u003e\n\u003cli\u003eBowman, S., Angeli, G., Potts, C., Manning, C. (2015). \u003ca href=\"https://arxiv.org/abs/1508.05326\"\u003eA Large Annotated Corpus for Learning Natural Language Inference\u003c/a\u003e. EMNLP.\u003c/li\u003e\n\u003cli\u003eWilliams, A., Nangia, N., Bowman, S. (2018). \u003ca href=\"https://arxiv.org/abs/1704.05426\"\u003eA Broad-Coverage Challenge Corpus for Sentence Understanding through Inference\u003c/a\u003e. NAACL.\u003c/li\u003e\n\u003cli\u003eZhang, T., Kishore, V., Wu, F., Weinberger, K., Artzi, Y. (2020). \u003ca href=\"https://arxiv.org/abs/1904.09675\"\u003eBERTScore: Evaluating Text Generation with BERT\u003c/a\u003e. ICLR.\u003c/li\u003e\n\u003cli\u003eZheng, L., et al. (2023). \u003ca href=\"https://arxiv.org/abs/2306.05685\"\u003eJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena\u003c/a\u003e. NeurIPS.\u003c/li\u003e\n\u003cli\u003eEs, S., James, J., Espinosa-Anke, L., Schockaert, S. (2024). \u003ca href=\"https://arxiv.org/abs/2309.15217\"\u003eRAGAS: Automated Evaluation of Retrieval Augmented Generation\u003c/a\u003e. EACL.\u003c/li\u003e\n\u003cli\u003ePanickssery, A., Bowman, S., Feng, S. (2024). \u003ca href=\"https://arxiv.org/abs/2404.13076\"\u003eLLM Evaluators Recognize and Favor Their Own Generations\u003c/a\u003e. NeurIPS.\u003c/li\u003e\n\u003c/ul\u003e\n","url":"http://gabelfra1.github.io/gabelfra.github.io/posts/evaluation/","date_published":"21066-21-09T60:2121:00+00:00","date_modified":"21066-21-09T60:2121:00+00:00","author":{"name":"Calvin Tran","url":"http://gabelfra1.github.io/gabelfra.github.io/"}},{"id":"52979df3e8a788f12c5eb1e21b29e5d60f1135ba","title":"PageIndex: When Your RAG Reads Like a Human","summary":"","content_text":"The Explainability Problem Nobody Talks About Standard RAG has a transparency problem, and it\u0026rsquo;s not one that shows up in benchmark numbers.\nWhen a user asks \u0026ldquo;why did the system return this paragraph and not that one?\u0026rdquo;, the honest answer is: \u0026ldquo;because the cosine similarity was 0.83 instead of 0.79.\u0026rdquo; That answer is technically correct and completely useless. Nobody outside of an ML team reasons about document retrieval in terms of vector distances. They reason about sections, chapters, topics, and arguments.\nThis gap between how retrieval systems work and how humans think about information lookup creates a problem: the users either accept the result blindly or reject the whole system because they can\u0026rsquo;t audit it. Neither is good.\nPageIndex is an approach that sidesteps this problem entirely by discarding vector similarity and replacing it with something more explainable : structure-aware, reasoning-based retrieval.\nHow Humans Actually Look Things Up If you remember searching for something in a physical library before Google, I do, you remember the process intuitively. You didn\u0026rsquo;t scan every page of every book. You:\nFound the right book by subject Opened the table of contents Found the relevant chapter Skimmed the section headings Read the specific paragraph That multi-level navigation was precise and fully explainable. You could tell anyone exactly why you landed on page 247 of Option futures and other derivatives.\nPageIndex replicates this exact behavior. Instead of embedding chunks and retrieving by similarity, it first builds a hierarchical index of the documents. Baiscally a machine-readable table of contents with summaries at each node and then asks an LLM to reason about which branch of that tree is likely to contain the answer. The LLM walks the index the same way I used to serarch the uni library.\nTwo Phases: Index Then Retrieve The workflow has two clean phases.\nPhase 1 — Build the index. Submit a document to PageIndex. It parses the document structure (headings, sections, subsections), generates a summary for each node, and returns a JSON tree. This happens once per document.\nPhase 2 — Reason and retrieve. At query time, pass the tree (without full text) to an LLM and ask it to identify which nodes are relevant. Then pull the actual text only from those nodes and generate the final answer.\nThe tree for a typical long document looks like this:\nDocument Root ├── Abstract [pages 1-1] │ └── \u0026#34;Overview of the proposed method and key results...\u0026#34; ├── Introduction [pages 2-3] │ └── \u0026#34;Motivation, problem statement, and prior work...\u0026#34; ├── Methodology [pages 4-8] │ ├── Data Collection [pages 4-5] │ ├── Model Architecture [pages 5-7] │ └── Training Procedure [pages 7-8] ├── Results [pages 9-12] │ ├── Quantitative Evaluation [pages 9-11] └── Conclusion [pages 11-12] └── \u0026#34;Summary of findings and future directions...\u0026#34; The LLM sees this structure with the summaries and can reason: \u0026ldquo;the question is about conclusions, so I should look at the Conclusion node.\u0026rdquo; That reasoning is visible, auditable, and directly explainable to any user.\nThe Code Setup is minimal : Clone, install dependencies, and set your LLM key. Then generate the tree locally from a PDF :\npython3 run_pageindex.py --pdf_path document.pdf \\ --if-add-node-summary yes \\ --if-add-node-text yes Load the result and set up your LLM client:\nimport pageindex.utils as utils import openai, json with open(\u0026#34;results/document_structure.json\u0026#34;) as f: tree = json.load(f) utils.print_tree(tree) def call_llm(prompt, model=\u0026#34;gpt-4.1\u0026#34;, temperature=0): client = openai.AsyncOpenAI() response = await client.chat.completions.create( model=model, messages=[{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: prompt}], temperature=temperature ) return response.choices[0].message.content.strip() At query time, strip the full text out of the tree, show it to the LLM, and ask for the relevant nodes:\nquery = \u0026#34;What are the main conclusions?\u0026#34; tree_without_text = utils.remove_fields(tree.copy(), fields=[\u0026#39;text\u0026#39;]) search_prompt = f\u0026#34;\u0026#34;\u0026#34; You are given a question and a tree structure of a document. Each node contains a node id, title, and summary. Find all nodes likely to contain the answer. Question: {query} Document tree: {json.dumps(tree_without_text, indent=2)} Reply in JSON: {{ \u0026#34;thinking\u0026#34;: \u0026#34;\u0026lt;your reasoning\u0026gt;\u0026#34;, \u0026#34;node_list\u0026#34;: [\u0026#34;node_id_1\u0026#34;, \u0026#34;node_id_2\u0026#34;] }} \u0026#34;\u0026#34;\u0026#34; tree_search_result = call_llm(search_prompt) Then extract text only from the selected nodes and generate the answer:\nnode_map = utils.create_node_mapping(tree) result = json.loads(tree_search_result) node_list = result[\u0026#34;node_list\u0026#34;] relevant_content = \u0026#34;\\n\\n\u0026#34;.join(node_map[nid][\u0026#34;text\u0026#34;] for nid in node_list) answer = await call_llm(f\u0026#34;Answer based on context:\\n\\nQuestion: {query}\\nContext: {relevant_content}\u0026#34;) Stateless by Design, Storable if Needed Because the index is just JSON, the whole system is flexible about state. You can run it entirely in memory for small queries, or persist the tree to disk, a database, or an object store, you do you.\nimport json # persist with open(\u0026#34;doc_tree.json\u0026#34;, \u0026#34;w\u0026#34;) as f: json.dump(tree, f) # restore with open(\u0026#34;doc_tree.json\u0026#34;) as f: tree = json.load(f) This is meaningful in practice. Vector databases introduce a whole infrastructure layer: embedding models, database extensions, fancy fusion algorithm. PageIndex\u0026rsquo;s index is a JSON file. It integrates with whatever you already have,an S3 bucket, a Redis cache,just some memory. No additional infrastructure required.\nStructure-Aware Chunking vs Fixed-Size Chunking This is perhaps the most underappreciated difference.\nTraditional RAG cuts documents into fixed-size chunks, typically 512 or 1024 tokens, with some overlap to avoid losing context. The problem is that documents are not uniformly structured. A single section might be 200 tokens; another might be 3000. A fixed-size chunk will routinely split a coherent argument mid-sentence or merge the end of one section with the start of an unrelated one.\nTraditional RAG chunking (fixed-size): │◄── 512 tokens ──►│◄── 512 tokens ──►│◄── 512 tokens ──►│ [intro...][ch1 end][ch2 start...mid ][...ch2 end][ch3...] PageIndex chunking (structure-aware): │◄── Introduction ──►│◄─────── Chapter 2 ─────────►│◄── Chapter 3 ──►│ [coherent unit ] [coherent unit ] [coherent unit ] Structure-aware chunks are semantically complete. When shown to the user as a source reference, they read like a passage — not like a sentence that got cut off because a token counter hit a limit.\nScaling Limitations PageIndex is not a silver bullet. It is fundamentally an in-context approach: the entire tree (without text) is passed to the LLM at query time. For very long documents with deep, granular structure, this tree can itself become large. The LLM must process it in a single context window.\nIt is worth being honest about this. But it is equally worth being honest about the fact that standard RAG also does not excel at scale. Top-k retrieval over large corpora produces noisy results; relevance degrades as the corpus grows and context bloating is real.\nFor the use cases where it fits: long structured documents, regulatory filings, technical manuals, research papers, contracts PageIndex is genuinely strong. FinanceBench results report ~98.7% accuracy on financial document QA.\nConclusion PageIndex is a good example of a retrieval approach that optimises for the right thing. Not just accuracy, but explainability. The retrieval path is a reasoning trace. The chunks are coherent sections. The index is a plain JSON file.\nNone of this is magic. It is essentially the same thing a well-organised student does when they open a textbook. The insight is just recognising that human information retrieval is already an excellent algorithm, and that replicating it is could be better than replacing it with cosine similarity.\nThe official repo and cookbook are at github.com/VectifyAI/PageIndex if you want to run the full notebook.\nAs a recovering statistician, I am profoundly happy when I can make an AI system a bit more explainable, and profoundly unhappy when I can\u0026rsquo;t.\n","content_html":"\u003ch3 id=\"the-explainability-problem-nobody-talks-about\"\u003eThe Explainability Problem Nobody Talks About\u003c/h3\u003e\n\u003cp\u003eStandard RAG has a transparency problem, and it\u0026rsquo;s not one that shows up in benchmark numbers.\u003c/p\u003e\n\u003cp\u003eWhen a user asks \u003cem\u003e\u0026ldquo;why did the system return this paragraph and not that one?\u0026rdquo;\u003c/em\u003e, the honest answer is: \u003cem\u003e\u0026ldquo;because the cosine similarity was 0.83 instead of 0.79.\u0026rdquo;\u003c/em\u003e That answer is technically correct and completely useless. Nobody outside of an ML team reasons about document retrieval in terms of vector distances. They reason about sections, chapters, topics, and arguments.\u003c/p\u003e\n\u003cp\u003eThis gap between how retrieval systems work and how humans think about information lookup creates a problem: the users either accept the result blindly or reject the whole system because they can\u0026rsquo;t audit it. Neither is good.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/VectifyAI/PageIndex\"\u003ePageIndex\u003c/a\u003e is an approach that sidesteps this problem entirely by discarding vector similarity and replacing it with something more explainable : \u003cstrong\u003estructure-aware, reasoning-based retrieval\u003c/strong\u003e.\u003c/p\u003e\n\u003chr\u003e\n\u003ch3 id=\"how-humans-actually-look-things-up\"\u003eHow Humans Actually Look Things Up\u003c/h3\u003e\n\u003cp\u003eIf you remember searching for something in a physical library before Google, I do, you remember the process intuitively. You didn\u0026rsquo;t scan every page of every book. You:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eFound the right book by subject\u003c/li\u003e\n\u003cli\u003eOpened the table of contents\u003c/li\u003e\n\u003cli\u003eFound the relevant chapter\u003c/li\u003e\n\u003cli\u003eSkimmed the section headings\u003c/li\u003e\n\u003cli\u003eRead the specific paragraph\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThat multi-level navigation was precise and fully explainable. You could tell anyone exactly why you landed on page 247 of \u003cem\u003eOption futures and other derivatives\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003ePageIndex replicates this exact behavior. Instead of embedding chunks and retrieving by similarity, it first builds a hierarchical index of the documents. Baiscally a machine-readable table of contents with summaries at each node and then asks an LLM to \u003cem\u003ereason\u003c/em\u003e about which branch of that tree is likely to contain the answer. The LLM walks the index the same way I used to serarch the uni library.\u003c/p\u003e\n\u003chr\u003e\n\u003ch3 id=\"two-phases-index-then-retrieve\"\u003eTwo Phases: Index Then Retrieve\u003c/h3\u003e\n\u003cp\u003eThe workflow has two clean phases.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePhase 1 — Build the index.\u003c/strong\u003e Submit a document to PageIndex.\nIt parses the document structure (headings, sections, subsections), generates a summary for each node, and returns a JSON tree. This happens once per document.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePhase 2 — Reason and retrieve.\u003c/strong\u003e At query time, pass the tree (without full text) to an LLM and ask it to identify which nodes are relevant. Then pull the actual text only from those nodes and generate the final answer.\u003c/p\u003e\n\u003cp\u003eThe tree for a typical long document looks like this:\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eDocument Root\n├── Abstract [pages 1-1]\n│   └── \u0026#34;Overview of the proposed method and key results...\u0026#34;\n├── Introduction [pages 2-3]\n│   └── \u0026#34;Motivation, problem statement, and prior work...\u0026#34;\n├── Methodology [pages 4-8]\n│   ├── Data Collection [pages 4-5]\n│   ├── Model Architecture [pages 5-7]\n│   └── Training Procedure [pages 7-8]\n├── Results [pages 9-12]\n│   ├── Quantitative Evaluation [pages 9-11]\n└── Conclusion [pages 11-12]\n    └── \u0026#34;Summary of findings and future directions...\u0026#34;\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eThe LLM sees this structure with the summaries and can reason: \u003cem\u003e\u0026ldquo;the question is about conclusions, so I should look at the Conclusion node.\u0026rdquo;\u003c/em\u003e That reasoning is visible, auditable, and directly explainable to any user.\u003c/p\u003e\n\u003chr\u003e\n\u003ch3 id=\"the-code\"\u003eThe Code\u003c/h3\u003e\n\u003cp\u003eSetup is minimal : Clone, install dependencies, and set your LLM key.\nThen generate the tree locally from a PDF :\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003epython3 run_pageindex.py --pdf_path document.pdf \u003cspan style=\"color:#8045ff\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#8045ff\"\u003e\u003c/span\u003e    --if-add-node-summary yes \u003cspan style=\"color:#8045ff\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#8045ff\"\u003e\u003c/span\u003e    --if-add-node-text yes\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eLoad the result and set up your LLM client:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003epageindex.utils\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eas\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eutils\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eopenai\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ejson\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003ewith\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eopen\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;results/document_structure.json\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eas\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003etree\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ejson\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eload\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eutils\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eprint_tree\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etree\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003ecall_llm\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eprompt\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;gpt-4.1\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etemperature\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eopenai\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eAsyncOpenAI\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eawait\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echat\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecompletions\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecreate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;user\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprompt\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003etemperature\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etemperature\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoices\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emessage\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econtent\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estrip\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eAt query time, strip the full text out of the tree, show it to the LLM, and ask for the relevant nodes:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003equery\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;What are the main conclusions?\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003etree_without_text\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eutils\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eremove_fields\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etree\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecopy\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efields\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;text\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003esearch_prompt\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003eYou are given a question and a tree structure of a document.\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003eEach node contains a node id, title, and summary.\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003eFind all nodes likely to contain the answer.\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003eQuestion: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003equery\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003eDocument tree: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ejson\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edumps\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etree_without_text\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eindent\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e2\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003eReply in JSON:\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003e\u003c/span\u003e\u003cspan style=\"color:#8045ff\"\u003e{{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003e    \u0026#34;thinking\u0026#34;: \u0026#34;\u0026lt;your reasoning\u0026gt;\u0026#34;,\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003e    \u0026#34;node_list\u0026#34;: [\u0026#34;node_id_1\u0026#34;, \u0026#34;node_id_2\u0026#34;]\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003e\u003c/span\u003e\u003cspan style=\"color:#8045ff\"\u003e}}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003etree_search_result\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecall_llm\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esearch_prompt\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThen extract text only from the selected nodes and generate the answer:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003enode_map\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eutils\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecreate_node_mapping\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etree\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eresult\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ejson\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eloads\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etree_search_result\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003enode_list\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresult\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;node_list\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003erelevant_content\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#8045ff\"\u003e\\n\\n\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ejoin\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003enode_map\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003enid\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e][\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;text\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enid\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enode_list\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eanswer\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eawait\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecall_llm\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Answer based on context:\u003c/span\u003e\u003cspan style=\"color:#8045ff\"\u003e\\n\\n\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003eQuestion: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003equery\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#8045ff\"\u003e\\n\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003eContext: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003erelevant_content\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003chr\u003e\n\u003ch3 id=\"stateless-by-design-storable-if-needed\"\u003eStateless by Design, Storable if Needed\u003c/h3\u003e\n\u003cp\u003eBecause the index is just JSON, the whole system is flexible about state. You can run it entirely in memory for small queries, or persist the tree to disk, a database, or an object store, you do you.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ejson\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# persist\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003ewith\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eopen\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;doc_tree.json\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;w\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eas\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ejson\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edump\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etree\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# restore\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003ewith\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eopen\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;doc_tree.json\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eas\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003etree\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ejson\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eload\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis is meaningful in practice. Vector databases introduce a whole infrastructure layer: embedding models, database extensions, fancy fusion algorithm. PageIndex\u0026rsquo;s index is a JSON file. It integrates with whatever you already have,an S3 bucket, a Redis cache,just some memory. No additional infrastructure required.\u003c/p\u003e\n\u003chr\u003e\n\u003ch3 id=\"structure-aware-chunking-vs-fixed-size-chunking\"\u003eStructure-Aware Chunking vs Fixed-Size Chunking\u003c/h3\u003e\n\u003cp\u003eThis is perhaps the most underappreciated difference.\u003c/p\u003e\n\u003cp\u003eTraditional RAG cuts documents into fixed-size chunks, typically 512 or 1024 tokens, with some overlap to avoid losing context. The problem is that documents are not uniformly structured. A single section might be 200 tokens; another might be 3000. A fixed-size chunk will routinely split a coherent argument mid-sentence or merge the end of one section with the start of an unrelated one.\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eTraditional RAG chunking (fixed-size):\n│◄── 512 tokens ──►│◄── 512 tokens ──►│◄── 512 tokens ──►│\n [intro...][ch1 end][ch2 start...mid  ][...ch2 end][ch3...]\n\nPageIndex chunking (structure-aware):\n│◄── Introduction ──►│◄─────── Chapter 2 ─────────►│◄── Chapter 3 ──►│\n [coherent unit       ] [coherent unit               ] [coherent unit   ]\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eStructure-aware chunks are semantically complete. When shown to the user as a source reference, they read like a passage — not like a sentence that got cut off because a token counter hit a limit.\u003c/p\u003e\n\u003chr\u003e\n\u003ch3 id=\"scaling-limitations\"\u003eScaling Limitations\u003c/h3\u003e\n\u003cp\u003ePageIndex is not a silver bullet. It is fundamentally an in-context approach: the entire tree (without text) is passed to the LLM at query time. For very long documents with deep, granular structure, this tree can itself become large. The LLM must process it in a single context window.\u003c/p\u003e\n\u003cp\u003eIt is worth being honest about this. But it is equally worth being honest  about the fact that \u003cstrong\u003estandard RAG also does not excel at scale\u003c/strong\u003e. Top-k retrieval over large corpora produces noisy results; relevance degrades as the corpus grows and context bloating is real.\u003c/p\u003e\n\u003cp\u003eFor the use cases where it fits: long structured documents, regulatory filings, technical manuals, research papers, contracts PageIndex is genuinely strong. \u003ca href=\"https://github.com/VectifyAI/PageIndex\"\u003eFinanceBench results\u003c/a\u003e report ~98.7% accuracy on financial document QA.\u003c/p\u003e\n\u003chr\u003e\n\u003ch3 id=\"conclusion\"\u003eConclusion\u003c/h3\u003e\n\u003cp\u003ePageIndex is a good example of a retrieval approach that optimises for the right thing. Not just accuracy, but \u003cstrong\u003eexplainability\u003c/strong\u003e. The retrieval path is a reasoning trace. The chunks are coherent sections. The index is a plain JSON file.\u003c/p\u003e\n\u003cp\u003eNone of this is magic. It is essentially the same thing a well-organised student does when they open a textbook. The insight is just recognising that human information retrieval is already an excellent algorithm, and that replicating it is could be better than replacing it with cosine similarity.\u003c/p\u003e\n\u003cp\u003eThe official repo and cookbook are at \u003ca href=\"https://github.com/VectifyAI/PageIndex\"\u003egithub.com/VectifyAI/PageIndex\u003c/a\u003e if you want to run the full notebook.\u003c/p\u003e\n\u003cp\u003eAs a recovering statistician, I am profoundly happy when I can make an AI system a bit more explainable, and profoundly unhappy when I can\u0026rsquo;t.\u003c/p\u003e\n","url":"http://gabelfra1.github.io/gabelfra.github.io/posts/pageindex/","date_published":"12056-12-09T50:1212:00+00:00","date_modified":"12056-12-09T50:1212:00+00:00","author":{"name":"Calvin Tran","url":"http://gabelfra1.github.io/gabelfra.github.io/"}},{"id":"cf596e0965fd916540d715f9485e6873272b1aa1","title":"Do Foundational Models Actually Work for Forecasting?","summary":"","content_text":"Traditional forecasting Every forecasting practitioner from a math/stats background knows the ritual. You get a new series, run the ADF test, differentiate, and hope that the series is stationary. You plot the ACF and PACF, count the significant lags, make a guess at p and q, run a grid search over AIC, wait, then, once you have figured out the period, you do it all over again with a seasonal component.\nFor the uninitiated, you can find a more in-depth explanation in the bible of forecasting: Forecasting: Principles and Practice\n# Step 1: determine d via ADF + KPSS def find_d(series, max_d=2): ser = series.copy() for d in range(max_d + 1): adf_p = adfuller(ser.dropna(), autolag=\u0026#34;AIC\u0026#34;)[1] kpss_p = kpss(ser.dropna(), regression=\u0026#34;c\u0026#34;, nlags=\u0026#34;auto\u0026#34;)[1] if (adf_p \u0026lt; 0.05) and (kpss_p \u0026gt; 0.05): return d ser = ser.diff() return max_d # Step 2: detect seasonal period from the periodogram s = detect_season(m4_series) # Step 3: AIC grid search over (p,q) x (P,Q) for (p, q), (P, Q) in itertools.product(search_pq, search_PQ): aic = SARIMAX(data, order=(p, d, q), seasonal_order=(P, D, Q, s)).fit(disp=False).aic ... If you wanted to modernize your forecasting approach and follow what practitioners on Kaggle are doing, you could switch to a machine learning model like LightGBM. However, with LightGBM, you first need to engineer lag features, decide how many lags to include, add rolling statistics, and tune hyperparameters such as num_leaves, learning_rate, and subsample using tools like Optuna before you can even think of generating a single forecast.\nstudy = optuna.create_study(direction=\u0026#34;minimize\u0026#34;, sampler=optuna.samplers.TPESampler(seed=42)) study.optimize(objective, n_trials=40, show_progress_bar=True) best_lgb_params = study.best_params The problem is worse than just tedium: none of this work transfers between projects. You accumulate better instincts and reusable code snippets, but every new series forces you to restart the entire process from scratch.\nDeep learning did not solve the problem The obvious hope was that deep learning would do what it did for images and text: learn a universal representation and make the hand-crafted pipeline obsolete. It did not work out that way.\nThe M4 competition (2018), covering 100,000 real-world series, was the first large-scale empirical test. The winner was a hybrid ES-RNN, not a pure deep learning model, but a classical Exponential Smoothing model with an RNN on top. Pure neural methods consistently underperformed compared to statistical baselines that had been around for decades.\nThe M5 and M6 competitions confirmed the pattern. The use of LightGBM with careful feature engineering repeatedly beat deep learning architectures. The community largely concluded that deep learning was simply too cumbersome to train on short series, too prone to overfitting, and too sensitive to architecture choices to be worth the cost of training.\nThe fundamental problem was that every deep learning model still needed to be trained from scratch on each dataset. It was essentially a more expensive version of the same ritual.\nWhat TimesFM changes Google Research published A decoder-only foundation model for time-series forecasting in 2023. I\u0026rsquo;ve noticed the advances in transformer models when they started to creep into the industry I work in, aka banking, but only recently, while reading the paper PRAGMA: Revolut Foundation Model, did it click. The core idea is simple: pre-train a large transformer on an enormous and diverse corpus of time series data, then use it zero-shot on any new series — no hyperparameter search, no statistical tests.\nTimesFM is to forecasting what GPT was to NLP: a single pre-trained model that generalises across tasks it has never seen.\nThis is the part that matters for practitioners. After all the machinery described above — stationarity tests, seasonal detection, AIC search, Optuna — the TimesFM inference path is:\nimport timesfm tfm = timesfm.TimesFm( hparams=timesfm.TimesFmHparams( backend=\u0026#34;torch\u0026#34;, horizon_len=14, context_len=512, ), checkpoint=timesfm.TimesFmCheckpoint( huggingface_repo_id=\u0026#34;google/timesfm-1.0-200m-pytorch\u0026#34; ), ) point_forecast, _ = tfm.forecast(inputs=[train_data], freq=[0]) There is no fit(). The model has never seen your series. You hand it the historical values, and it returns a forecast. The entire statistical/ML ritual is replaced by a forward pass through a pre-trained network.\nRunning the same backtesting as SARIMA and LightGBM To make this a fair comparison, all three models were evaluated on M4 Daily series D2047 — 8,533 daily observations — using identical TimeSeriesSplit cross-validation with five folds and a 14-step forecast horizon.\nfrom sklearn.model_selection import TimeSeriesSplit n_splits = 5 test_size = 14 tscv = TimeSeriesSplit(n_splits=n_splits, test_size=test_size) # SARIMA: fit a new model per fold with the tuned (p,d,q)(P,D,Q,s) for fold, (train_idx, test_idx) in enumerate(tscv.split(m4_series)): fit = SARIMAX(train_data, order=best_order, seasonal_order=best_sorder).fit(disp=False) preds = fit.forecast(steps=test_size).values # LightGBM: train a new model per fold with Optuna-tuned params for fold, (train_idx, test_idx) in enumerate(tscv.split(m4_series)): model = lgb.LGBMRegressor(**best_lgb_params) model.fit(X_tr, y_tr) # recursive prediction ... # TimesFM: no training, same context each fold for fold, (train_idx, test_idx) in enumerate(tscv.split(m4_series)): point_forecast, _ = tfm.forecast(inputs=[train_data], freq=[0]) preds = point_forecast[0] SARIMA requires running the full stationarity and order-selection pipeline before the loop. LightGBM requires several Optuna trials on an inner cross-validation. TimesFM requires none of that — the same three-line setup serves every fold unchanged.\nThe result: TimesFM achieves competitive MAE without a single line of training code, while SARIMA and LightGBM each needed a bespoke calibration pipeline just to participate.\nWhy this could democratise forecasting TimesFM collapses the forecasting stack. A developer with no background in time-series statistics can load the model, pass in a dataframe, and get a probabilistic forecast within seconds.\nThis mirrors exactly what happened to NLP. Before the transformer era, building a production sentiment classifier for customer service calls meant curating internal transcriptions, collecting customer feedback labels, engineering linguistic features, selecting and tuning a model, and repeating this process for every new business domain or language. After GPT, it is possible to load a pre-trained model, pass in your call transcripts, and get sentiment predictions without annotation pipelines. The barrier to entry collapsed by an order of magnitude, and the volume of NLP applications in production exploded.\nThe limits are real I didn\u0026rsquo;t spend five years studying statistical theory and another five as a data scientist just to watch it all get dismissed. Time series is weird in ways that GPT\u0026rsquo;s text never had to deal with. Financial returns don\u0026rsquo;t behave like energy demand. Sensor data from a factory floor look nothing like web traffic. The distribution shift between Google\u0026rsquo;s pre-training corpus and your specific problem can be absolutely massive.\nFine-tuning on domain-specific data can help close much of that gap, and Google\u0026rsquo;s own benchmarks show that even a few hundred in-domain examples significantly improve TimesFM\u0026rsquo;s accuracy. But that is still a fundamentally different burden than training SARIMA from scratch — it is adaptation, not construction.\nConclusion Over the past decade, the pattern in machine learning has been consistent: provided that a large enough pre-training corpus is available, foundation models eventually outperform or at the very least match task-specific models trained from scratch for a fraction of the deployment cost. Text, images, audio, code — the story is always the same.\nTime series proved more resistant simply because the data is messier, the frequency and domain vary wildly, and the benchmarks were designed around the assumption that every model would be trained from scratch. TimesFM, and the family of models following it, challenge this core assumption.\nWhether TimesFM specifically becomes the standard, or is surpassed by the next generation of temporal foundation models, the direction is now clear. The era of bespoke per-series models is ending.\n","content_html":"\u003ch3 id=\"traditional-forecasting\"\u003eTraditional forecasting\u003c/h3\u003e\n\u003cp\u003eEvery forecasting practitioner from a math/stats background knows the ritual.\nYou get a new series, run the ADF test, differentiate, and hope that the series is stationary. You plot the ACF and PACF, count the significant lags, make a guess at \u003ccode\u003ep\u003c/code\u003e and \u003ccode\u003eq\u003c/code\u003e, run a grid search over AIC, wait, then, once you have figured out the period, you do it all over again with a seasonal component.\u003c/p\u003e\n\u003cp\u003eFor the uninitiated, you can find a more in-depth explanation in the bible of forecasting:\n\u003ca href=\"https://otexts.com/fpp3/\"\u003eForecasting: Principles and Practice\u003c/a\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# Step 1: determine d via ADF + KPSS\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003efind_d\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eseries\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emax_d\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e2\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eser\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eseries\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecopy\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ed\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erange\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emax_d\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e+\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003eadf_p\u003c/span\u003e  \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eadfuller\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eser\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edropna\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eautolag\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;AIC\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003ekpss_p\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ekpss\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eser\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edropna\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eregression\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;c\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enlags\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;auto\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eadf_p\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e\u0026lt;\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e0.05\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eand\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ekpss_p\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e0.05\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ed\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003eser\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eser\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ediff\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emax_d\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# Step 2: detect seasonal period from the periodogram\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003es\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edetect_season\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003em4_series\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# Step 3: AIC grid search over (p,q) x (P,Q)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ep\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eq\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eQ\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eitertools\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eproduct\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esearch_pq\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esearch_PQ\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eaic\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eSARIMAX\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edata\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eorder\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ep\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ed\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eq\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                  \u003cspan style=\"color:#111\"\u003eseasonal_order\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eP\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eD\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eQ\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003es\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edisp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eFalse\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eaic\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#f92672\"\u003e...\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eIf you wanted to modernize your forecasting approach and follow what practitioners on Kaggle are doing, you could switch to a machine learning model like LightGBM. However, with LightGBM, you first need to engineer lag features, decide how many lags to include, add rolling statistics, and tune hyperparameters such as \u003ccode\u003enum_leaves\u003c/code\u003e, \u003ccode\u003elearning_rate\u003c/code\u003e, and \u003ccode\u003esubsample\u003c/code\u003e using tools like Optuna before you can even think of generating a single forecast.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003estudy\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eoptuna\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecreate_study\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edirection\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;minimize\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                            \u003cspan style=\"color:#111\"\u003esampler\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eoptuna\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esamplers\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eTPESampler\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eseed\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e42\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003estudy\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eoptimize\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eobjective\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003en_trials\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e40\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eshow_progress_bar\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eTrue\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003ebest_lgb_params\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estudy\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ebest_params\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe problem is worse than just tedium: \u003cstrong\u003enone of this work transfers between projects\u003c/strong\u003e. You accumulate better instincts and reusable code snippets, but every new series forces you to restart the entire process from scratch.\u003c/p\u003e\n\u003ch3 id=\"deep-learning-did-not-solve-the-problem\"\u003eDeep learning did not solve the problem\u003c/h3\u003e\n\u003cp\u003eThe obvious hope was that deep learning would do what it did for images and text: learn a universal representation and make the hand-crafted pipeline obsolete. It did not work out that way.\u003c/p\u003e\n\u003cp\u003eThe M4 competition (2018), covering 100,000 real-world series, was the first large-scale empirical test. The winner was a hybrid ES-RNN, not a pure deep learning model, but a classical Exponential Smoothing model with an RNN on top. Pure neural methods consistently underperformed compared to statistical baselines that had been around for decades.\u003c/p\u003e\n\u003cp\u003eThe M5 and M6 competitions confirmed the pattern. The use of LightGBM with careful feature engineering repeatedly beat deep learning architectures.\nThe community largely concluded that deep learning was simply too cumbersome to train on short series, too prone to overfitting, and too sensitive to architecture choices to be worth the cost of training.\u003c/p\u003e\n\u003cp\u003eThe fundamental problem was that every deep learning model still needed to be trained from scratch on each dataset. It was essentially a more expensive version of the same ritual.\u003c/p\u003e\n\u003ch3 id=\"what-timesfm-changes\"\u003eWhat TimesFM changes\u003c/h3\u003e\n\u003cp\u003eGoogle Research published \u003ca href=\"https://arxiv.org/abs/2310.10688\"\u003eA decoder-only foundation model for time-series forecasting\u003c/a\u003e in 2023.\nI\u0026rsquo;ve noticed the advances in transformer models when they started to creep into the industry I work in, aka banking, but only recently, while reading the paper \u003ca href=\"https://arxiv.org/html/2604.08649v1\"\u003ePRAGMA: Revolut Foundation Model\u003c/a\u003e, did it click.\nThe core idea is simple: pre-train a large transformer on an enormous and diverse corpus of time series data, then use it zero-shot on any new series — no hyperparameter search, no statistical tests.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTimesFM is to forecasting what GPT was to NLP: a single pre-trained model that generalises across tasks it has never seen.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis is the part that matters for practitioners. After all the machinery described above — stationarity tests, seasonal detection, AIC search, Optuna — the TimesFM inference path is:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etimesfm\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003etfm\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etimesfm\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eTimesFm\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ehparams\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etimesfm\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eTimesFmHparams\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003ebackend\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;torch\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003ehorizon_len\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e14\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003econtext_len\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e512\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003echeckpoint\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etimesfm\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eTimesFmCheckpoint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003ehuggingface_repo_id\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;google/timesfm-1.0-200m-pytorch\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003epoint_forecast\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e_\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etfm\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eforecast\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003einputs\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etrain_data\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efreq\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThere is no \u003ccode\u003efit()\u003c/code\u003e. The model has never seen your series. You hand it the historical values, and it returns a forecast. The entire statistical/ML ritual is replaced by a forward pass through a pre-trained network.\u003c/p\u003e\n\u003ch3 id=\"running-the-same-backtesting-as-sarima-and-lightgbm\"\u003eRunning the same backtesting as SARIMA and LightGBM\u003c/h3\u003e\n\u003cp\u003eTo make this a fair comparison, all three models were evaluated on M4 Daily series D2047 — 8,533 daily observations — using identical \u003ccode\u003eTimeSeriesSplit\u003c/code\u003e cross-validation with five folds and a 14-step forecast horizon.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esklearn.model_selection\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eTimeSeriesSplit\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003en_splits\u003c/span\u003e  \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e5\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003etest_size\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e14\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003etscv\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eTimeSeriesSplit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003en_splits\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003en_splits\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etest_size\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etest_size\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# SARIMA: fit a new model per fold with the tuned (p,d,q)(P,D,Q,s)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efold\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etrain_idx\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etest_idx\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eenumerate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etscv\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esplit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003em4_series\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003efit\u003c/span\u003e   \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eSARIMAX\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etrain_data\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eorder\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ebest_order\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                    \u003cspan style=\"color:#111\"\u003eseasonal_order\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ebest_sorder\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edisp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eFalse\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003epreds\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efit\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eforecast\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esteps\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etest_size\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003evalues\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# LightGBM: train a new model per fold with Optuna-tuned params\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efold\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etrain_idx\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etest_idx\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eenumerate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etscv\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esplit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003em4_series\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elgb\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eLGBMRegressor\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e**\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ebest_lgb_params\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eX_tr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ey_tr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#75715e\"\u003e# recursive prediction ...\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# TimesFM: no training, same context each fold\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efold\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etrain_idx\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etest_idx\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eenumerate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etscv\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esplit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003em4_series\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003epoint_forecast\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e_\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etfm\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eforecast\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003einputs\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etrain_data\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efreq\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003epreds\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003epoint_forecast\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eSARIMA requires running the full stationarity and order-selection pipeline before the loop. LightGBM requires several Optuna trials on an inner cross-validation. TimesFM requires none of that — the same three-line setup serves every fold unchanged.\u003c/p\u003e\n\u003cp\u003eThe result: \u003cstrong\u003eTimesFM achieves competitive MAE without a single line of training code\u003c/strong\u003e, while SARIMA and LightGBM each needed a bespoke calibration pipeline just to participate.\u003c/p\u003e\n\u003cfigure\u003e\n  \u003cimg src=\"../../images/timesfm_cv_mae.png\" alt=\"time_series_cv\"\u003e\n\u003c/figure\u003e\n\u003ch3 id=\"why-this-could-democratise-forecasting\"\u003eWhy this could democratise forecasting\u003c/h3\u003e\n\u003cp\u003eTimesFM collapses the forecasting stack. A developer with no background in time-series statistics can load the model, pass in a dataframe, and get a probabilistic forecast within seconds.\u003c/p\u003e\n\u003cp\u003eThis mirrors exactly what happened to NLP. Before the transformer era, building a production sentiment classifier for customer service calls meant curating internal transcriptions, collecting customer feedback labels, engineering linguistic features, selecting and tuning a model, and repeating this process for every new business domain or language. After GPT, it is possible to load a pre-trained model, pass in your call transcripts, and get sentiment predictions without annotation pipelines. The barrier to entry collapsed by an order of magnitude, and the volume of NLP applications in production exploded.\u003c/p\u003e\n\u003ch3 id=\"the-limits-are-real\"\u003eThe limits are real\u003c/h3\u003e\n\u003cp\u003eI didn\u0026rsquo;t spend five years studying statistical theory and another five as a data scientist just to watch it all get dismissed. Time series is \u003cem\u003eweird\u003c/em\u003e in ways that GPT\u0026rsquo;s text never had to deal with. Financial returns don\u0026rsquo;t behave like energy demand. Sensor data from a factory floor look nothing like web traffic. The distribution shift between Google\u0026rsquo;s pre-training corpus and your specific problem can be absolutely massive.\u003c/p\u003e\n\u003cp\u003eFine-tuning on domain-specific data can help close much of that gap, and Google\u0026rsquo;s own benchmarks show that even a few hundred in-domain examples significantly improve TimesFM\u0026rsquo;s accuracy. But that is still a fundamentally different burden than training SARIMA from scratch — it is adaptation, not construction.\u003c/p\u003e\n\u003ch3 id=\"conclusion\"\u003eConclusion\u003c/h3\u003e\n\u003cp\u003eOver the past decade, the pattern in machine learning has been consistent: provided that a large enough pre-training corpus is available, foundation models eventually outperform or at the very least match task-specific models trained from scratch for a fraction of the deployment cost. Text, images, audio, code — the story is always the same.\u003c/p\u003e\n\u003cp\u003eTime series proved more resistant simply because the data is messier, the frequency and domain vary wildly, and the benchmarks were designed around the assumption that every model would be trained from scratch. TimesFM, and the family of models following it, challenge this core assumption.\u003c/p\u003e\n\u003cp\u003eWhether TimesFM specifically becomes the standard, or is surpassed by the next generation of temporal foundation models, the direction is now clear. The era of bespoke per-series models is ending.\u003c/p\u003e\n","url":"http://gabelfra1.github.io/gabelfra.github.io/posts/timesfm/","date_published":"18046-18-09T40:1818:00+00:00","date_modified":"18046-18-09T40:1818:00+00:00","author":{"name":"Calvin Tran","url":"http://gabelfra1.github.io/gabelfra.github.io/"}},{"id":"6d5ca05fa7bf86d1462f79539b3a800ed03db774","title":"Building effective agents","summary":"","content_text":"Agentic pattern We\u0026rsquo;re doing something different today. Instead of a typical article, this is a code along tutorial based on Anthropic\u0026rsquo;s foundational blogpost on agentic patterns : Building effective agents.\nThe objective of this article is to demystify the complexity of agentic workflows by presenting a simple and clear Python implementation of the core concepts defined in the Anthropic article.\n0) Set-up Replicating this exercise is possible with any LLM provider that is compatible with the Openai Client library and the set-up is minimal. We just need the LLM client, pydantic for structured output and json to parse the response.\n# --- Libraries --- import openai from pydantic import BaseModel from pydantic import BaseModel, Field import json from typing import Literal, TypedDict 1) Prompt chaining Prompt chaining decomposes a task into a sequence of steps, where each LLM call processes the output of the previous one.\nIn this simple example we are going to create a chain (sequence) of prompts (generate_joke,improve_joke,polish_joke) in order to generate an output (a joke).\nThe chain represents the simplest workflow pattern, marking the first step in complexity beyond standard one-shot prompting for generating answers.\n# --- State --- state = { \u0026#34;topic\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;joke\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;improved_joke\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;final_joke\u0026#34;: \u0026#34;\u0026#34; } # --- Functions --- def generate_joke(topic: str) -\u0026gt; str: response = client.chat.completions.create( messages=[{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: f\u0026#34;Write a joke about {topic} with just single a sentence\u0026#34;}], ) return response.choices[0].message.content def improve_joke(joke: str) -\u0026gt; str: response = client.chat.completions.create( messages=[{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: f\u0026#34;Make this joke funnier by adding wordplay or puns: {joke}\u0026#34;}], ) return response.choices[0].message.content def polish_joke(joke: str) -\u0026gt; str: response = client.chat.completions.create( messages=[{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: f\u0026#34;\u0026#34;\u0026#34; Write a short, funny joke in the following format: Why did [subject] [action]? To [funny reason related to the subject]! The joke should be based on the following : {joke} \u0026#34;\u0026#34;\u0026#34;}], ) return response.choices[0].message.content def check_punchline(joke: str) -\u0026gt; bool: return joke.count(\u0026#34;?\u0026#34;) \u0026gt;= 1 # --- Workflow Execution --- def run_workflow(topic: str): state[\u0026#34;topic\u0026#34;] = topic state[\u0026#34;joke\u0026#34;] = generate_joke(topic) state[\u0026#34;improved_joke\u0026#34;] = improve_joke(state[\u0026#34;joke\u0026#34;]) state[\u0026#34;final_joke\u0026#34;] = polish_joke(state[\u0026#34;improved_joke\u0026#34;]) if check_punchline(state[\u0026#34;final_joke\u0026#34;]): print(state[\u0026#34;final_joke\u0026#34;]) else : print(\u0026#34;joke did not passed quality gate\u0026#34;) return state # --- Run --- result = run_workflow(\u0026#34;cats at work\u0026#34;) 2) Routing Routing classifies an input and directs it to a specialized followup task.\nIn this example, we use an initial LLM call to determine the next step whether to generate a poem, a story, or a joke, and then execute only the selected route.\nThe routing pattern is particularly interesting because it leverages specialization in prompts and data sources. Instead of having one generalist \u0026lsquo;agent\u0026rsquo; that knows everything, it is more efficient to employ multiple specialized agents that the router can direct tasks to as needed.\n# --- Schema for routing --- class Route(BaseModel): step: Literal[\u0026#34;poem\u0026#34;, \u0026#34;story\u0026#34;, \u0026#34;joke\u0026#34;] = Field( None, description=\u0026#34;The next step in the routing process\u0026#34; ) # --- State --- class State(TypedDict): input: str decision: str output: str # --- Router using structured output --- def llm_call_router(state: State) -\u0026gt; dict: response = client.chat.completions.create( messages=[ {\u0026#34;role\u0026#34;: \u0026#34;system\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;Route the input to story, joke, or poem based on the user\u0026#39;s request.\u0026#34;}, {\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: state[\u0026#34;input\u0026#34;]}, ], response_format={ \u0026#34;type\u0026#34;: \u0026#34;json_schema\u0026#34;, \u0026#34;json_schema\u0026#34;: { \u0026#34;name\u0026#34;: \u0026#34;route\u0026#34;, \u0026#34;schema\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: { \u0026#34;step\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;enum\u0026#34;: [\u0026#34;poem\u0026#34;, \u0026#34;story\u0026#34;, \u0026#34;joke\u0026#34;] } }, \u0026#34;required\u0026#34;: [\u0026#34;step\u0026#34;] } } }, ) decision = json.loads(response.choices[0].message.content) return {\u0026#34;decision\u0026#34;: decision[\u0026#34;step\u0026#34;]} # --- Nodes --- def llm_call_story(state: State) -\u0026gt; dict: response = client.chat.completions.create( messages=[{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: f\u0026#34;Write a story about: {state[\u0026#39;input\u0026#39;]}\u0026#34;}], ) return {\u0026#34;output\u0026#34;: response.choices[0].message.content} def llm_call_joke(state: State) -\u0026gt; dict: response = client.chat.completions.create( messages=[{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: f\u0026#34;Write a joke about: {state[\u0026#39;input\u0026#39;]}\u0026#34;}], ) return {\u0026#34;output\u0026#34;: response.choices[0].message.content} def llm_call_poem(state: State) -\u0026gt; dict: response = client.chat.completions.create( messages=[{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: f\u0026#34;Write a poem about: {state[\u0026#39;input\u0026#39;]}\u0026#34;}], ) return {\u0026#34;output\u0026#34;: response.choices[0].message.content} # --- Routing Logic --- def route_decision(state: State) -\u0026gt; str: if state[\u0026#34;decision\u0026#34;] == \u0026#34;story\u0026#34;: return \u0026#34;story\u0026#34; elif state[\u0026#34;decision\u0026#34;] == \u0026#34;joke\u0026#34;: return \u0026#34;joke\u0026#34; elif state[\u0026#34;decision\u0026#34;] == \u0026#34;poem\u0026#34;: return \u0026#34;poem\u0026#34; # --- Workflow Execution --- def run_workflow(user_input: str): state: State = {\u0026#34;input\u0026#34;: user_input, \u0026#34;decision\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;output\u0026#34;: \u0026#34;\u0026#34;} # Step 1: Route route_result = llm_call_router(state) state[\u0026#34;decision\u0026#34;] = route_result[\u0026#34;decision\u0026#34;] # Step 2: Execute based on decision if state[\u0026#34;decision\u0026#34;] == \u0026#34;story\u0026#34;: state.update(llm_call_story(state)) elif state[\u0026#34;decision\u0026#34;] == \u0026#34;joke\u0026#34;: state.update(llm_call_joke(state)) elif state[\u0026#34;decision\u0026#34;] == \u0026#34;poem\u0026#34;: state.update(llm_call_poem(state)) return state # --- Run --- result = run_workflow(\u0026#34;Write me a short story about cats at work\u0026#34;) 3) Parallelization Breaking a task into independent subtasks run in parallel.\nUnlike the previous example, which decided between a poem, a story, or a joke, this case involves generating all three in parallel and then aggregating the results.\nThis is a typical pattern across computer science: when a problem is too complex or slow for a single computation, it is broken down into simpler ones to achieve greater efficiency and reduced processing time.\n# --- State --- class State(TypedDict): topic: str joke: str story: str poem: str combined_output: str # --- LLM Call Functions --- def call_llm(prompt: str) -\u0026gt; str: response = client.chat.completions.create( messages=[{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: prompt}], ) return response.choices[0].message.content def call_llm_1(state: State) -\u0026gt; dict: return {\u0026#34;joke\u0026#34;: call_llm(f\u0026#34;Write a joke about {state[\u0026#39;topic\u0026#39;]}\u0026#34;)} def call_llm_2(state: State) -\u0026gt; dict: return {\u0026#34;story\u0026#34;: call_llm(f\u0026#34;Write a story about {state[\u0026#39;topic\u0026#39;]}\u0026#34;)} def call_llm_3(state: State) -\u0026gt; dict: return {\u0026#34;poem\u0026#34;: call_llm(f\u0026#34;Write a poem about {state[\u0026#39;topic\u0026#39;]}\u0026#34;)} def aggregator(state: State) -\u0026gt; dict: combined = f\u0026#34;Here\u0026#39;s a story, joke, and poem about {state[\u0026#39;topic\u0026#39;]}!\\n\\n\u0026#34; combined += f\u0026#34;STORY:\\n{state[\u0026#39;story\u0026#39;]}\\n\\n\u0026#34; combined += f\u0026#34;JOKE:\\n{state[\u0026#39;joke\u0026#39;]}\\n\\n\u0026#34; combined += f\u0026#34;POEM:\\n{state[\u0026#39;poem\u0026#39;]}\u0026#34; return {\u0026#34;combined_output\u0026#34;: combined} # --- Workflow Execution --- def run_workflow(topic: str) -\u0026gt; State: state: State = {\u0026#34;topic\u0026#34;: topic, \u0026#34;joke\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;story\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;poem\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;combined_output\u0026#34;: \u0026#34;\u0026#34;} # Run tasks in parallel (or sequentially for simplicity) state.update(call_llm_1(state)) state.update(call_llm_2(state)) state.update(call_llm_3(state)) # Aggregate results state.update(aggregator(state)) return state # --- Run --- result = run_workflow(\u0026#34;cats\u0026#34;) 4) Evaluator-optimizer In the evaluator-optimizer workflow, one LLM call generates a response while another provides evaluation and feedback in a loop.\nIn this example, we implement a feedback loop: the evaluator grades the generated joke and returns feedback to the generator until the resulting joke is deemed satisfactory.\nThis is a highly interesting pattern as it represents a significant step toward true autonomy. In this case, two \u0026lsquo;agents\u0026rsquo; collaboratively \u0026rsquo;think\u0026rsquo; about a problem until they reach consensus that it is solved, operating without any human intervention or feedback.\nclass State(TypedDict): joke: str topic: str feedback: str funny_or_not: str # --- Schema for evaluation --- def llm_call_evaluator(state: State) -\u0026gt; dict: response = client.chat.completions.create( messages=[ {\u0026#34;role\u0026#34;: \u0026#34;system\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;Grade the joke as funny or not funny and provide feedback if needed.\u0026#34;}, {\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: f\u0026#34;Joke: {state[\u0026#39;joke\u0026#39;]}\u0026#34;} ], response_format={ \u0026#34;type\u0026#34;: \u0026#34;json_schema\u0026#34;, \u0026#34;json_schema\u0026#34;: { \u0026#34;name\u0026#34;: \u0026#34;feedback\u0026#34;, \u0026#34;schema\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: { \u0026#34;grade\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;enum\u0026#34;: [\u0026#34;funny\u0026#34;, \u0026#34;not funny\u0026#34;]}, \u0026#34;feedback\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;} }, \u0026#34;required\u0026#34;: [\u0026#34;grade\u0026#34;, \u0026#34;feedback\u0026#34;] } } }, ) result = json.loads(response.choices[0].message.content) return {\u0026#34;funny_or_not\u0026#34;: result[\u0026#34;grade\u0026#34;], \u0026#34;feedback\u0026#34;: result[\u0026#34;feedback\u0026#34;]} # --- Joke Generator --- def llm_call_generator(state: State) -\u0026gt; dict: if state.get(\u0026#34;feedback\u0026#34;): prompt = f\u0026#34;Write a joke about {state[\u0026#39;topic\u0026#39;]} but improve it based on this feedback: {state[\u0026#39;feedback\u0026#39;]}\u0026#34; else: prompt = f\u0026#34;Write a joke about {state[\u0026#39;topic\u0026#39;]}\u0026#34; response = client.chat.completions.create( messages=[{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: prompt}], ) return {\u0026#34;joke\u0026#34;: response.choices[0].message.content} # --- Workflow --- def run_workflow(topic: str, max_loops: int = 5): state: State = {\u0026#34;topic\u0026#34;: topic, \u0026#34;joke\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;feedback\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;funny_or_not\u0026#34;: \u0026#34;\u0026#34;} for i in range(max_loops): # Generate joke state.update(llm_call_generator(state)) # Evaluate joke eval_result = llm_call_evaluator(state) state.update(eval_result) if state[\u0026#34;funny_or_not\u0026#34;] == \u0026#34;funny\u0026#34; and i\u0026gt;0 : print(\u0026#34; -- Joke accepted! --\u0026#34;) break else: print(\u0026#34;Joke rejected, improving...\u0026#34;) return state # --- Run --- result = run_workflow(\u0026#34;cats at work\u0026#34;) 5) Agents Agents are emerging in key capabilities—understanding complex inputs, engaging in reasoning and planning, using tools reliably, and recovering from errors.\nFinally, we demonstrate an example of a truly autonomous agent. Though limited to arithmetic tools in this case, the agent can receive a task, reason, plan which tools to use, execute those tools, and autonomously stop the reasoning process when it is ready to generate the final answer to the user.\n# --- Define Python functions --- def multiply(a: int, b: int) -\u0026gt; int: return a * b def add(a: int, b: int) -\u0026gt; int: return a + b def divide(a: int, b: int) -\u0026gt; float: return a / b # --- Define tool schemas --- tools = [ { \u0026#34;name\u0026#34;: \u0026#34;multiply\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Multiply two integers\u0026#34;, \u0026#34;parameters\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: { \u0026#34;a\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;integer\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;First integer\u0026#34;}, \u0026#34;b\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;integer\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Second integer\u0026#34;} }, \u0026#34;required\u0026#34;: [\u0026#34;a\u0026#34;, \u0026#34;b\u0026#34;] } }, { \u0026#34;name\u0026#34;: \u0026#34;add\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Add two integers\u0026#34;, \u0026#34;parameters\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: { \u0026#34;a\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;integer\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;First integer\u0026#34;}, \u0026#34;b\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;integer\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Second integer\u0026#34;} }, \u0026#34;required\u0026#34;: [\u0026#34;a\u0026#34;, \u0026#34;b\u0026#34;] } }, { \u0026#34;name\u0026#34;: \u0026#34;divide\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Divide two integers\u0026#34;, \u0026#34;parameters\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: { \u0026#34;a\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;integer\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;First integer\u0026#34;}, \u0026#34;b\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;integer\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Second integer\u0026#34;} }, \u0026#34;required\u0026#34;: [\u0026#34;a\u0026#34;, \u0026#34;b\u0026#34;] } } ] # --- Map tool names to Python functions --- tools_by_name = { \u0026#34;multiply\u0026#34;: multiply, \u0026#34;add\u0026#34;: add, \u0026#34;divide\u0026#34;: divide } import json # --- Workflow --- def run_agent(user_input: str, max_iterations: int = 5): messages = [ {\u0026#34;role\u0026#34;: \u0026#34;system\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;You are a helpful assistant tasked with performing arithmetic on a set of inputs.\u0026#34;}, {\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: user_input} ] for i in range(max_iterations): response = client.chat.completions.create( model=model, messages=messages, tools=tools, tool_choice=\u0026#34;auto\u0026#34; ) last_message = response.choices[0].message tool_calls = last_message.tool_calls # Keep the last LLM message in history messages.append({\u0026#34;role\u0026#34;: \u0026#34;assistant\u0026#34;, \u0026#34;content\u0026#34;: last_message.content, \u0026#34;tool_calls\u0026#34;: tool_calls}) if tool_calls: print(f\u0026#34;\\n Iteration {i+1}: Tool calls detected\u0026#34;) # Execute tool calls for call in tool_calls: func_name = call.function.name args = json.loads(call.function.arguments) # Print tool name and arguments print(f\u0026#34; Calling tool: {func_name} with arguments: {args}\u0026#34;) result = tools_by_name[func_name](**args) # Append tool response messages.append({ \u0026#34;role\u0026#34;: \u0026#34;tool\u0026#34;, \u0026#34;tool_call_id\u0026#34;: call.id, \u0026#34;content\u0026#34;: str(result) }) else: # No tool calls → final answer print(f\u0026#34;\\nFinal answer after {i+1} iterations: {last_message.content}\u0026#34;) return last_message.content return \u0026#34;Max iterations reached without a final answer.\u0026#34; # --- Run --- answer = run_agent(\u0026#34;Please multiply 3 and 4 and divide the result by 2\u0026#34;) Conclusion I hope these examples have helped the reader realise the simplicity behind the patterns for implementing agentic workflows. While using higher-level packages like LangChain offers \u0026lsquo;shorter\u0026rsquo; implementation methods, this often obscures the underlying details. To truly understand these core concepts, I believe a simpler approach using just the API and basic Python is most effective.\n","content_html":"\u003ch3 id=\"agentic-pattern\"\u003eAgentic pattern\u003c/h3\u003e\n\u003cp\u003eWe\u0026rsquo;re doing something different today.\nInstead of a typical article, this is a code along tutorial based on Anthropic\u0026rsquo;s foundational blogpost on agentic patterns : \u003ca href=\"https://www.anthropic.com/engineering/building-effective-agents\"\u003eBuilding effective agents\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eThe objective of this article is to demystify the complexity of agentic workflows by presenting a simple and clear Python implementation of the core concepts defined in the Anthropic article.\u003c/p\u003e\n\u003ch3 id=\"0-set-up\"\u003e0) Set-up\u003c/h3\u003e\n\u003cp\u003eReplicating this exercise is possible with any LLM provider that is compatible with the \u003ca href=\"http://platform.openai.com/docs/libraries\"\u003eOpenai Client library\u003c/a\u003e and the set-up is minimal.\nWe just need the LLM client, pydantic for \u003ca href=\"https://ai.pydantic.dev/output/\"\u003estructured output\u003c/a\u003e and json to parse the response.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Libraries ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eopenai\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003epydantic\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eBaseModel\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003epydantic\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eBaseModel\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eField\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ejson\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etyping\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eLiteral\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eTypedDict\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"1-prompt-chaining\"\u003e1) Prompt chaining\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003ePrompt chaining decomposes a task into a sequence of steps, where each LLM call processes the output of the previous one\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eIn this simple example we are going to create a chain (sequence) of prompts (generate_joke,improve_joke,polish_joke) in order to generate an output (a joke).\u003c/p\u003e\n\u003cp\u003eThe chain represents the simplest workflow pattern, marking the first step in complexity beyond standard one-shot prompting for generating answers.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- State ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;topic\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;improved_joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;final_joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Functions ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003egenerate_joke\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etopic\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echat\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecompletions\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecreate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;user\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Write a joke about \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etopic\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e with just single a sentence\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoices\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emessage\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econtent\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eimprove_joke\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ejoke\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echat\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecompletions\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecreate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;user\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Make this joke funnier by adding wordplay or puns: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ejoke\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoices\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emessage\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econtent\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003epolish_joke\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ejoke\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echat\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecompletions\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecreate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;user\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003e                Write a short, funny joke in the following format:\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003e                Why did [subject] [action]?\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003e                To [funny reason related to the subject]!\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003e                The joke should be based on the following : \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ejoke\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003e                \u0026#34;\u0026#34;\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoices\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emessage\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econtent\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003echeck_punchline\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ejoke\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ebool\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ejoke\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecount\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;?\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e\u0026gt;=\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Workflow Execution ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003erun_workflow\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etopic\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;topic\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etopic\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003egenerate_joke\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etopic\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;improved_joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eimprove_joke\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;final_joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003epolish_joke\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;improved_joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003echeck_punchline\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;final_joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;final_joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003eelse\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;joke did not passed quality gate\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Run ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eresult\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erun_workflow\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;cats at work\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"2-routing\"\u003e2) Routing\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eRouting classifies an input and directs it to a specialized followup task\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eIn this example, we use an initial LLM call to determine the next step whether to generate a poem, a story, or a joke, and then execute only the selected route.\u003c/p\u003e\n\u003cp\u003eThe routing pattern is particularly interesting because it leverages specialization in prompts and data sources. Instead of having one generalist \u0026lsquo;agent\u0026rsquo; that knows everything, it is more efficient to employ multiple specialized agents that the router can direct tasks to as needed.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Schema for routing ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003eclass\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eRoute\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eBaseModel\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003estep\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eLiteral\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;poem\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;story\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eField\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003eNone\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edescription\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;The next step in the routing process\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- State ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003eclass\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eState\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eTypedDict\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003einput\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003edecision\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eoutput\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Router using structured output ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003ellm_call_router\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eState\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edict\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echat\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecompletions\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecreate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;system\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Route the input to story, joke, or poem based on the user\u0026#39;s request.\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;user\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;input\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003eresponse_format\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;type\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;json_schema\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;json_schema\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#d88200\"\u003e\u0026#34;name\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;route\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#d88200\"\u003e\u0026#34;schema\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;type\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;object\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;properties\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;step\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;type\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;string\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;enum\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;poem\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;story\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                        \u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                    \u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;required\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;step\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003edecision\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ejson\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eloads\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoices\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emessage\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econtent\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;decision\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edecision\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;step\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Nodes ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003ellm_call_story\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eState\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edict\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echat\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecompletions\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecreate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;user\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Write a story about: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;input\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;output\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoices\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emessage\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econtent\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003ellm_call_joke\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eState\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edict\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echat\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecompletions\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecreate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;user\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Write a joke about: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;input\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;output\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoices\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emessage\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econtent\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003ellm_call_poem\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eState\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edict\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echat\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecompletions\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecreate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;user\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Write a poem about: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;input\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;output\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoices\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emessage\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econtent\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Routing Logic ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eroute_decision\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eState\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;decision\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;story\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;story\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003eelif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;decision\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;joke\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003eelif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;decision\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;poem\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;poem\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Workflow Execution ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003erun_workflow\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003euser_input\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eState\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;input\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003euser_input\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;decision\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;output\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#75715e\"\u003e# Step 1: Route\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eroute_result\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ellm_call_router\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;decision\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eroute_result\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;decision\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#75715e\"\u003e# Step 2: Execute based on decision\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;decision\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;story\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eupdate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ellm_call_story\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003eelif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;decision\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eupdate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ellm_call_joke\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003eelif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;decision\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;poem\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eupdate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ellm_call_poem\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Run ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eresult\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erun_workflow\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Write me a short story about cats at work\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"3-parallelization\"\u003e3) Parallelization\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eBreaking a task into independent subtasks run in parallel\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eUnlike the previous example, which decided between a poem, a story, or a joke, this case involves generating all three in parallel and then aggregating the results.\u003c/p\u003e\n\u003cp\u003eThis is a typical pattern across computer science: when a problem is too complex or slow for a single computation, it is broken down into simpler ones to achieve greater efficiency and reduced processing time.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- State ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003eclass\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eState\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eTypedDict\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003etopic\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ejoke\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003estory\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003epoem\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ecombined_output\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- LLM Call Functions ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003ecall_llm\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eprompt\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echat\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecompletions\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecreate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;user\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprompt\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoices\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emessage\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econtent\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003ecall_llm_1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eState\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edict\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecall_llm\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Write a joke about \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;topic\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003ecall_llm_2\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eState\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edict\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;story\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecall_llm\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Write a story about \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;topic\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003ecall_llm_3\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eState\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edict\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;poem\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecall_llm\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Write a poem about \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;topic\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eaggregator\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eState\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edict\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ecombined\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Here\u0026#39;s a story, joke, and poem about \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;topic\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e!\u003c/span\u003e\u003cspan style=\"color:#8045ff\"\u003e\\n\\n\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ecombined\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e+=\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;STORY:\u003c/span\u003e\u003cspan style=\"color:#8045ff\"\u003e\\n\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;story\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#8045ff\"\u003e\\n\\n\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ecombined\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e+=\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;JOKE:\u003c/span\u003e\u003cspan style=\"color:#8045ff\"\u003e\\n\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;joke\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#8045ff\"\u003e\\n\\n\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ecombined\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e+=\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;POEM:\u003c/span\u003e\u003cspan style=\"color:#8045ff\"\u003e\\n\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;poem\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;combined_output\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecombined\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Workflow Execution ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003erun_workflow\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etopic\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eState\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eState\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;topic\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etopic\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;story\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;poem\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;combined_output\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#75715e\"\u003e# Run tasks in parallel (or sequentially for simplicity)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eupdate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecall_llm_1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eupdate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecall_llm_2\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eupdate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecall_llm_3\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#75715e\"\u003e# Aggregate results\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eupdate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eaggregator\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Run ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eresult\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erun_workflow\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;cats\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"4-evaluator-optimizer\"\u003e4) Evaluator-optimizer\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eIn the evaluator-optimizer workflow, one LLM call generates a response while another provides evaluation and feedback in a loop\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eIn this example, we implement a feedback loop: the evaluator grades the generated joke and returns feedback to the generator until the resulting joke is deemed satisfactory.\u003c/p\u003e\n\u003cp\u003eThis is a highly interesting pattern as it represents a significant step toward true autonomy. In this case, two \u0026lsquo;agents\u0026rsquo; collaboratively \u0026rsquo;think\u0026rsquo; about a problem until they reach consensus that it is solved, operating without any human intervention or feedback.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003eclass\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eState\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eTypedDict\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ejoke\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003etopic\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003efeedback\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003efunny_or_not\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Schema for evaluation ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003ellm_call_evaluator\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eState\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edict\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echat\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecompletions\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecreate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;system\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Grade the joke as funny or not funny and provide feedback if needed.\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;user\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Joke: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;joke\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003eresponse_format\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;type\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;json_schema\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;json_schema\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#d88200\"\u003e\u0026#34;name\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;feedback\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#d88200\"\u003e\u0026#34;schema\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;type\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;object\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;properties\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;grade\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;type\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;string\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;enum\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;funny\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;not funny\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;feedback\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;type\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;string\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                    \u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;required\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;grade\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;feedback\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eresult\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ejson\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eloads\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoices\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emessage\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econtent\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;funny_or_not\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresult\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;grade\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;feedback\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresult\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;feedback\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Joke Generator ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003ellm_call_generator\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eState\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edict\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eget\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;feedback\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003eprompt\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Write a joke about \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;topic\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e but improve it based on this feedback: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;feedback\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003eelse\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003eprompt\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Write a joke about \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;topic\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echat\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecompletions\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecreate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;user\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprompt\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoices\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emessage\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econtent\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Workflow ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003erun_workflow\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etopic\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emax_loops\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e5\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eState\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;topic\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etopic\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;joke\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;feedback\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;funny_or_not\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erange\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emax_loops\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#75715e\"\u003e# Generate joke\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eupdate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ellm_call_generator\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#75715e\"\u003e# Evaluate joke\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003eeval_result\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ellm_call_evaluator\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eupdate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eeval_result\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;funny_or_not\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;funny\u0026#34;\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eand\u003c/span\u003e  \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e\u0026gt;\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34; -- Joke accepted! --\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#00a8c8\"\u003ebreak\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003eelse\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Joke rejected, improving...\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estate\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Run ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eresult\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erun_workflow\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;cats at work\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"5-agents\"\u003e5) Agents\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eAgents are emerging in key capabilities—understanding complex inputs, engaging in reasoning and planning, using tools reliably, and recovering from errors\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eFinally, we demonstrate an example of a truly autonomous agent. Though limited to arithmetic tools in this case, the agent can receive a task, reason, plan which tools to use, execute those tools, and autonomously stop the reasoning process when it is ready to generate the final answer to the user.\u003c/p\u003e\n\u003cfigure\u003e\n  \u003cimg src=\"../../images/anthropic.png\" \n  alt=\"agentic_pattern\"\u003e\n\u003c/figure\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Define Python functions ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003emultiply\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ea\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eb\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ea\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e*\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eb\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eadd\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ea\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eb\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ea\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e+\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eb\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003edivide\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ea\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eb\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efloat\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ea\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e/\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eb\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Define tool schemas ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003etools\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;name\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;multiply\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;description\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Multiply two integers\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;parameters\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;type\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;object\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;properties\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#d88200\"\u003e\u0026#34;a\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;type\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;integer\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;description\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;First integer\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#d88200\"\u003e\u0026#34;b\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;type\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;integer\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;description\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Second integer\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;required\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;a\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;b\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;name\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;add\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;description\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Add two integers\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;parameters\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;type\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;object\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;properties\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#d88200\"\u003e\u0026#34;a\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;type\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;integer\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;description\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;First integer\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#d88200\"\u003e\u0026#34;b\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;type\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;integer\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;description\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Second integer\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;required\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;a\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;b\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;name\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;divide\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;description\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Divide two integers\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#d88200\"\u003e\u0026#34;parameters\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;type\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;object\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;properties\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#d88200\"\u003e\u0026#34;a\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;type\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;integer\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;description\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;First integer\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#d88200\"\u003e\u0026#34;b\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;type\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;integer\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;description\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Second integer\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;required\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;a\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;b\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Map tool names to Python functions ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003etools_by_name\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;multiply\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emultiply\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;add\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eadd\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;divide\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edivide\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ejson\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Workflow ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003erun_agent\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003euser_input\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emax_iterations\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eint\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e5\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;system\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;You are a helpful assistant tasked with performing arithmetic on a set of inputs.\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;user\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003euser_input\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erange\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emax_iterations\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echat\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecompletions\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecreate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003etools\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etools\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003etool_choice\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;auto\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003elast_message\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresponse\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoices\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emessage\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003etool_calls\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elast_message\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etool_calls\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#75715e\"\u003e# Keep the last LLM message in history\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eappend\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e({\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;assistant\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elast_message\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econtent\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;tool_calls\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etool_calls\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e})\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etool_calls\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#8045ff\"\u003e\\n\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e Iteration \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e+\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e: Tool calls detected\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#75715e\"\u003e# Execute tool calls\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecall\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etool_calls\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#111\"\u003efunc_name\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecall\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efunction\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#111\"\u003eargs\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ejson\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eloads\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecall\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efunction\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003earguments\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#75715e\"\u003e# Print tool name and arguments\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34; Calling tool: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efunc_name\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e with arguments: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eargs\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#111\"\u003eresult\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etools_by_name\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efunc_name\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e](\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e**\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eargs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#75715e\"\u003e# Append tool response\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eappend\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e({\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;tool\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;tool_call_id\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecall\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eid\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estr\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eresult\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#111\"\u003e})\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003eelse\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#75715e\"\u003e# No tool calls → final answer\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#8045ff\"\u003e\\n\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003eFinal answer after \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e+\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e iterations: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elast_message\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econtent\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elast_message\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econtent\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;Max iterations reached without a final answer.\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# --- Run ---\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eanswer\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003erun_agent\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Please multiply 3 and 4 and divide the result by 2\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"conclusion\"\u003eConclusion\u003c/h3\u003e\n\u003cp\u003eI hope these examples have helped the reader realise the simplicity behind the patterns for implementing agentic workflows. While using higher-level packages like \u003ca href=\"https://docs.langchain.com/oss/python/langgraph/workflows-agents\"\u003eLangChain\u003c/a\u003e offers \u0026lsquo;shorter\u0026rsquo; implementation methods, this often obscures the underlying details. To truly understand these core concepts, I believe a simpler approach using just the API and basic Python is most effective.\u003c/p\u003e\n","url":"http://gabelfra1.github.io/gabelfra.github.io/posts/agentic/","date_published":"10126-10-09T120:1010:00+00:00","date_modified":"10126-10-09T120:1010:00+00:00","author":{"name":"Calvin Tran","url":"http://gabelfra1.github.io/gabelfra.github.io/"}},{"id":"49dec6bff41bb1b7481f598115e8be42abc16c67","title":"LLM are way too confident","summary":"","content_text":"The Paradox of High Accuracy and Low Reliability In this article, we\u0026rsquo;ll explore LLM\u0026rsquo;s impressive ability to achieve high zero-shot performance on a classification tasks and at the same time being way too overconfident on all of their responses.\nCalibration it ensures that the model \u0026ldquo;knows what it doesn\u0026rsquo;t know.\u0026rdquo; For instance, if a model assigns a probability of 70% to a collection of predictions, we expect that 70% of those specific predictions will be correct in reality. When a model is perfectly calibrated, you can directly trust its confidence scores as an honest statement of its likelihood of being right.\nGPT can easily classify a sample without any prior fine-tuning but its confidence scores often don\u0026rsquo;t accurately reflect the likelihood of its predictions being correct. This disparity between performance and reliability is a significant challenge for deploying these models in real-world applications.\nThe data The dataset for this article is sourced from scikit-learn (sklearn) open datasets. This is a classic NLP classification problem where the raw data consists of various news articles, and the target variable maps to 20 distinct article topics.\nFor the sake of simplifying this demonstration, we filtered the original corpus. Our reduced dataset now exclusively comprises articles belonging to three specific categories: \u0026lsquo;mac\u0026rsquo;, \u0026lsquo;motorcycles\u0026rsquo;, and \u0026lsquo;baseball\u0026rsquo;.\nfrom sklearn.datasets import fetch_20newsgroups categories_to_fetch = [\u0026#39;comp.sys.mac.hardware\u0026#39;,\u0026#39;rec.motorcycles\u0026#39;,\u0026#39;rec.sport.baseball\u0026#39;] simplified_names = [\u0026#39;mac\u0026#39;,\u0026#39;motor\u0026#39;,\u0026#39;baseball\u0026#39;] # Fetch the data, removing headers/footers/quotes for cleaner text newsgroups_train = fetch_20newsgroups( subset=\u0026#39;train\u0026#39;, remove=(\u0026#39;headers\u0026#39;, \u0026#39;footers\u0026#39;, \u0026#39;quotes\u0026#39;), categories=categories_to_fetch, shuffle=True, random_state=42 ) newsgroups_test = fetch_20newsgroups( subset=\u0026#39;test\u0026#39;, remove=(\u0026#39;headers\u0026#39;, \u0026#39;footers\u0026#39;, \u0026#39;quotes\u0026#39;), categories=categories_to_fetch, shuffle=True, random_state=42 ) The baseline model Before the AI model, we first need to establish a baseline classifier. This was accomplished using a classical machine learning approach: we vectorized the text data using the well-established TF-IDF (Term Frequency-Inverse Document Frequency) method, and then trained a simple Naive Bayes classifier on the resulting features.\nfrom sklearn.feature_extraction.text import TfidfVectorizer from sklearn.naive_bayes import MultinomialNB vectorizer = TfidfVectorizer() vectors = vectorizer.fit_transform(newsgroups_train.data) clf = MultinomialNB(alpha=.01) clf.fit(vectors, newsgroups_train.target) vectors_test = vectorizer.transform(newsgroups_test.data) pred = clf.predict(vectors_test) Our Naive Bayes baseline model achieved a respectable 85% F1 score. Crucially, it also demonstrated a decently calibrated output:\nThe AI model Now we move to a modern approach for this NLP task: zero-shot classification. By leveraging the OpenAI API we completely bypass the need for any training. Instead, we tap directly into the vast pre-training knowledge of GPT and use a prompt to instruct the model to classify the news articles. This method allows us to achieve high performance without a single training loop.\nThis Python snippet defines a function, get_api_metrics_data, which performs zero-shot classification on a given news article using the OpenAI GPT-4o-mini model. It uses a system prompt to enforce classification into one of three specified categories (simplified_names) and is configured to return log probabilities for the top tokens (logprobs=True, top_logprobs=3) OpenAI LogProb cookbook.\nfrom openai import OpenAI # Prompt SYSTEM_PROMPT = f\u0026#34;\u0026#34;\u0026#34;You are an expert classifier. Classify the article into exactly one of the following categories: {\u0026#39;, \u0026#39;.join(simplified_names)}. Return ONLY the category name, exactly as it appears in the list, and nothing else.\u0026#34;\u0026#34;\u0026#34; # Inference function def get_api_metrics_data(article): \u0026#34;\u0026#34;\u0026#34; Calls the API once and returns both the probability vector (for calibration) and the predicted integer ID. \u0026#34;\u0026#34;\u0026#34; prob_vector = np.zeros(3) pred_id = -1 try: completion = client.chat.completions.create( model=\u0026#34;gpt-4o-mini\u0026#34;, messages=[{\u0026#34;role\u0026#34;: \u0026#34;system\u0026#34;, \u0026#34;content\u0026#34;: SYSTEM_PROMPT}, {\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: f\u0026#34;Article: {article}...\\nCategory:\u0026#34;}], logprobs=True, top_logprobs=3, temperature=0, ) predicted_name = completion.choices[0].message.content.strip().lower() pred_id = category_to_id.get(predicted_name, -1) logprobs_content = completion.choices[0].logprobs.content if logprobs_content: first_token_logprobs = logprobs_content[0].top_logprobs class_probs = np.zeros(3) for lp_item in first_token_logprobs: token = lp_item.token.strip().lower() linear_prob = np.exp(lp_item.logprob) if \u0026#39;mac\u0026#39; in token or token == \u0026#39;mac\u0026#39;: class_probs[0] += linear_prob elif \u0026#39;motor\u0026#39; in token or token == \u0026#39;motor\u0026#39;: class_probs[1] += linear_prob elif \u0026#39;baseball\u0026#39; in token or token == \u0026#39;baseball\u0026#39;: class_probs[2] += linear_prob total_relevant_prob = np.sum(class_probs) if total_relevant_prob \u0026gt; 0: prob_vector = class_probs / total_relevant_prob else: if pred_id != -1: prob_vector[pred_id] = 1.0 return prob_vector, pred_id except Exception: return np.zeros(3), -1 Our AI model demonstrated an impressive F1 score of 95%, significantly outperforming the Naive Bayes baseline by a solid 10% margin. However the model exhibits severe miscalibration, oscillating between being overconfident when its predictions are likely wrong, and being overly pessimistic (underconfident) when its predictions are correct :\nCan we fix it ? To address the observed miscalibration, we applied a traditional post-processing technique: fitting a second regression model to map the estimated scores to true probabilities. Specifically, we leveraged the Isotonic regression from the scikit-learn package, a common method for recalibrating uncalibrated estimators.\nHowever, isolating the results for a specific class, such as the \u0026lsquo;hardware\u0026rsquo; (mac) category, clearly shows the limitations of this approach:\nAs the initial confidence scores from the GPT model exhibited virtually no correlation with the real class probabilities, it proved impossible to fix the miscalibration effectively using Isotonic Regression or any other simple linear or non-linear transformation. The underlying issue is the fundamental instability of the model\u0026rsquo;s confidence scores.\nWhy is this a problem ? In most practical applications, simply providing a prediction label is insufficient; a clear measure of the confidence level in that prediction is absolutely essential. Consider a fraud detection system: it is designed to automatically approve transactions with high-confidence predictions of safety, while low-confidence predictions are deliberately flagged for mandatory human review. If the model\u0026rsquo;s underlying confidence scores are inaccurate or \u0026ldquo;miscalibrated,\u0026rdquo; this entire risk-managed workflow completely breaks down, leading to either high-risk transactions being overlooked or excessive manual review of safe transactions.\nConclusion While Large Language Models (LLMs) like GPT offer incredible zero-shot performance on NLP tasks, completely eliminating the need for training data, they suffer from a fundamental practical limitation: they currently lack an effective and reliable mechanism for measuring output uncertainty and calibration.\nThis absence means that in any use case where decision-making critically depends on a trustworthy measure of confidence or risk, modern LLMs become a terrible choice. Instead, practitioners should rely on simpler, well-calibrated Machine Learning models, even though these require the initial investment of collecting and training on labeled data. The superior performance of the LLM isn\u0026rsquo;t always worth the cost of unreliable risk assessment.\n","content_html":"\u003ch3 id=\"the-paradox-of-high-accuracy-and-low-reliability\"\u003eThe Paradox of High Accuracy and Low Reliability\u003c/h3\u003e\n\u003cp\u003eIn this article, we\u0026rsquo;ll explore LLM\u0026rsquo;s impressive ability to achieve \u003ca href=\"https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf\"\u003ehigh zero-shot performance on a classification tasks\u003c/a\u003e and at the same time being way too \u003ca href=\"https://arxiv.org/abs/2505.02151\"\u003eoverconfident\u003c/a\u003e on all of their responses.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCalibration\u003c/strong\u003e it ensures that the model \u0026ldquo;knows what it doesn\u0026rsquo;t know.\u0026rdquo;\nFor instance, if a model assigns a probability of 70% to a collection of predictions, we expect that 70% of those specific predictions will be correct in reality. When a model is perfectly calibrated, you can directly trust its confidence scores as an honest statement of its likelihood of being right.\u003c/p\u003e\n\u003cp\u003eGPT can easily classify a sample without any prior fine-tuning but its confidence scores often don\u0026rsquo;t accurately reflect the likelihood of its predictions being correct. This disparity between performance and reliability is a significant challenge for deploying these models in real-world applications.\u003c/p\u003e\n\u003ch4 id=\"the-data\"\u003eThe data\u003c/h4\u003e\n\u003cp\u003eThe dataset for this article is sourced from scikit-learn (sklearn) open datasets. This is a classic NLP classification problem where the raw data consists of various news articles, and the target variable maps to 20 distinct article topics.\u003c/p\u003e\n\u003cp\u003eFor the sake of simplifying this demonstration, we filtered the original corpus. Our reduced dataset now exclusively comprises articles belonging to three specific categories: \u0026lsquo;mac\u0026rsquo;, \u0026lsquo;motorcycles\u0026rsquo;, and \u0026lsquo;baseball\u0026rsquo;.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esklearn.datasets\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efetch_20newsgroups\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003ecategories_to_fetch\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;comp.sys.mac.hardware\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;rec.motorcycles\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;rec.sport.baseball\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003esimplified_names\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;mac\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;motor\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;baseball\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# Fetch the data, removing headers/footers/quotes for cleaner text\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003enewsgroups_train\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efetch_20newsgroups\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003esubset\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;train\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eremove\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;headers\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#39;footers\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#39;quotes\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ecategories\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecategories_to_fetch\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eshuffle\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eTrue\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003erandom_state\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e42\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003enewsgroups_test\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efetch_20newsgroups\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003esubset\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;test\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eremove\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;headers\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#39;footers\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#39;quotes\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ecategories\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecategories_to_fetch\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eshuffle\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eTrue\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003erandom_state\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e42\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch4 id=\"the-baseline-model\"\u003eThe baseline model\u003c/h4\u003e\n\u003cp\u003eBefore the AI model, we first need to establish a baseline classifier. This was accomplished using a classical machine learning approach: we vectorized the text data using the well-established TF-IDF (Term Frequency-Inverse Document Frequency) method, and then trained a simple Naive Bayes classifier on the resulting features.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esklearn.feature_extraction.text\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eTfidfVectorizer\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esklearn.naive_bayes\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eMultinomialNB\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003evectorizer\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eTfidfVectorizer\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003evectors\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003evectorizer\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efit_transform\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003enewsgroups_train\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edata\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eclf\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eMultinomialNB\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ealpha\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e.01\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eclf\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efit\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003evectors\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enewsgroups_train\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etarget\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003evectors_test\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003evectorizer\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etransform\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003enewsgroups_test\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edata\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003epred\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclf\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003epredict\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003evectors_test\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eOur Naive Bayes baseline model achieved a respectable \u003cstrong\u003e85% F1 score\u003c/strong\u003e.\nCrucially, it also demonstrated a decently calibrated output:\u003c/p\u003e\n\u003cfigure\u003e\n  \u003cimg src=\"../../images/baseline_calibration.png\" alt=\"baseline_calibration\"\u003e\n\u003c/figure\u003e\n\u003ch4 id=\"the-ai-model\"\u003eThe AI model\u003c/h4\u003e\n\u003cp\u003eNow we move to a modern approach for this NLP task: \u003cstrong\u003ezero-shot classification\u003c/strong\u003e.\nBy leveraging the \u003ca href=\"https://openai.com/index/gpt-4-research/\"\u003eOpenAI API\u003c/a\u003e we completely bypass the need for any training. Instead, we tap directly into the vast pre-training knowledge of GPT and use a prompt to instruct the model to classify the news articles. This method allows us to achieve high performance without a single training loop.\u003c/p\u003e\n\u003cp\u003eThis Python snippet defines a function, get_api_metrics_data, which performs zero-shot classification on a given news article using the OpenAI GPT-4o-mini model. It uses a system prompt to enforce classification into one of three specified categories (simplified_names) and is configured to return log probabilities for the top tokens (logprobs=True, top_logprobs=3) \u003ca href=\"https://cookbook.openai.com/examples/using_logprobs\"\u003eOpenAI LogProb cookbook\u003c/a\u003e.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eopenai\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eOpenAI\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# Prompt \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eSYSTEM_PROMPT\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u0026#34;You are an expert classifier.\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003eClassify the article into exactly one of the following categories: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#39;, \u0026#39;\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ejoin\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esimplified_names\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e.\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003eReturn ONLY the category name, exactly as it appears in the list, and nothing else.\u0026#34;\u0026#34;\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# Inference function\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eget_api_metrics_data\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003earticle\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u0026#34;\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003e    Calls the API once and returns both the probability vector (for calibration) \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003e    and the predicted integer ID.\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#d88200\"\u003e    \u0026#34;\u0026#34;\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eprob_vector\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ezeros\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e3\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e \n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003epred_id\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003etry\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003ecompletion\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclient\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echat\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecompletions\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecreate\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;gpt-4o-mini\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003emessages\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;system\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eSYSTEM_PROMPT\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e},\u003c/span\u003e \n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                      \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;role\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;user\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;content\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;Article: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003earticle\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e...\u003c/span\u003e\u003cspan style=\"color:#8045ff\"\u003e\\n\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003eCategory:\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e}],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003elogprobs\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eTrue\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003etop_logprobs\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e3\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003etemperature\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003epredicted_name\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecompletion\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoices\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emessage\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econtent\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estrip\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elower\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003epred_id\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecategory_to_id\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eget\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003epredicted_name\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003elogprobs_content\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ecompletion\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003echoices\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elogprobs\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003econtent\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elogprobs_content\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003efirst_token_logprobs\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elogprobs_content\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etop_logprobs\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003eclass_probs\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ezeros\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e3\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elp_item\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003efirst_token_logprobs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#111\"\u003etoken\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elp_item\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etoken\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estrip\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elower\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#111\"\u003elinear_prob\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eexp\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elp_item\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elogprob\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#39;mac\u0026#39;\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etoken\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etoken\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#39;mac\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                    \u003cspan style=\"color:#111\"\u003eclass_probs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e+=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elinear_prob\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#00a8c8\"\u003eelif\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#39;motor\u0026#39;\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etoken\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etoken\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#39;motor\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                    \u003cspan style=\"color:#111\"\u003eclass_probs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e+=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elinear_prob\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#00a8c8\"\u003eelif\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#39;baseball\u0026#39;\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etoken\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etoken\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#39;baseball\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                    \u003cspan style=\"color:#111\"\u003eclass_probs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e2\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e+=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elinear_prob\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003etotal_relevant_prob\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esum\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eclass_probs\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etotal_relevant_prob\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e0\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#111\"\u003eprob_vector\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eclass_probs\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e/\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etotal_relevant_prob\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#00a8c8\"\u003eelse\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003epred_id\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e!=\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e                    \u003cspan style=\"color:#111\"\u003eprob_vector\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003epred_id\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e1.0\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eprob_vector\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003epred_id\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003eexcept\u003c/span\u003e \u003cspan style=\"color:#75af00\"\u003eException\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#111\"\u003enp\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ezeros\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e3\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e),\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e-\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e \n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eOur AI model demonstrated an impressive \u003cstrong\u003eF1 score of 95%\u003c/strong\u003e, significantly outperforming the Naive Bayes baseline by a solid 10% margin.\nHowever the model exhibits \u003cstrong\u003esevere miscalibration\u003c/strong\u003e, oscillating between being overconfident when its predictions are likely wrong, and being overly pessimistic (underconfident) when its predictions are correct :\u003c/p\u003e\n\u003cfigure\u003e\n  \u003cimg src=\"../../images/ai_calibration.png\" alt=\"baseline_calibration\"\u003e\n\u003c/figure\u003e\n\u003ch4 id=\"can-we-fix-it-\"\u003eCan we fix it ?\u003c/h4\u003e\n\u003cp\u003eTo address the observed miscalibration, we applied a traditional post-processing technique: fitting a second regression model to map the estimated scores to true probabilities. Specifically, we leveraged the \u003ca href=\"https://scikit-learn.org/stable/modules/calibration.html\"\u003eIsotonic regression\u003c/a\u003e from the scikit-learn package, a common method for recalibrating uncalibrated estimators.\u003c/p\u003e\n\u003cp\u003eHowever, isolating the results for a specific class, such as the \u0026lsquo;hardware\u0026rsquo; (mac) category, clearly shows the limitations of this approach:\u003c/p\u003e\n\u003cfigure\u003e\n  \u003cimg src=\"../../images/openai_calibration_plot_isotonic_mac.png\" alt=\"baseline_calibration\"\u003e\n\u003c/figure\u003e\n\u003cp\u003eAs the initial confidence scores from the GPT model exhibited virtually no correlation with the real class probabilities, it proved impossible to fix the miscalibration effectively using Isotonic Regression or any other simple linear or non-linear transformation. The underlying issue is the fundamental instability of the model\u0026rsquo;s confidence scores.\u003c/p\u003e\n\u003ch4 id=\"why-is-this-a-problem-\"\u003eWhy is this a problem ?\u003c/h4\u003e\n\u003cp\u003eIn most practical applications, simply providing a prediction label is insufficient; a clear measure of the confidence level in that prediction is absolutely essential. Consider a fraud detection system: it is designed to automatically approve transactions with high-confidence predictions of safety, while low-confidence predictions are deliberately flagged for mandatory human review. If the model\u0026rsquo;s underlying confidence scores are inaccurate or \u0026ldquo;miscalibrated,\u0026rdquo; this entire risk-managed workflow completely breaks down, leading to either high-risk transactions being overlooked or excessive manual review of safe transactions.\u003c/p\u003e\n\u003ch3 id=\"conclusion\"\u003eConclusion\u003c/h3\u003e\n\u003cp\u003eWhile Large Language Models (LLMs) like GPT offer incredible zero-shot performance on NLP tasks, completely eliminating the need for training data, they suffer from a fundamental practical limitation: \u003cstrong\u003ethey currently lack an effective and reliable mechanism for measuring output uncertainty and calibration\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eThis absence means that in any use case where decision-making critically depends on a trustworthy measure of confidence or risk, modern LLMs become a terrible choice. Instead, practitioners should rely on simpler, well-calibrated Machine Learning models, even though these require the initial investment of collecting and training on labeled data. The superior performance of the LLM isn\u0026rsquo;t always  worth the cost of unreliable risk assessment.\u003c/p\u003e\n","url":"http://gabelfra1.github.io/gabelfra.github.io/posts/calibration/","date_published":"11106-11-09T100:1111:00+00:00","date_modified":"11106-11-09T100:1111:00+00:00","author":{"name":"Calvin Tran","url":"http://gabelfra1.github.io/gabelfra.github.io/"}},{"id":"e22a2002f294f8fba617f91e1885c7cd34f8b24e","title":"Diarization \u0026 Transcription with Whisper and PyAnnote","summary":"","content_text":"Speaker Diarization answers the question \u0026ldquo;who spoke when?\u0026rdquo; by segmenting an audio stream based on speaker identity, while transcription tells us \u0026ldquo;what was said\u0026rdquo;.\nIntroducing the Key Components Whisper: Automatic Speech Recognition model that converts speech to text. Pyannote.audio: Deep neural network for speaker diarization. WhisperX: Optimized Whisper variant that integrates multiple components. A Step-by-Step Implementation Guide This section provides a practical guide to implementing speaker diarization using WhisperX, Pyannote, and Whisper, while highlighting WhisperX\u0026rsquo;s advantages. For our audio sample, we can grab a segment from pretty much any free podcast out there – let\u0026rsquo;s go for a clip from Lex Friedman, one of my personal favorites.\nSetup Before diving into the code, it is essential to prepare your development environment. Make sure you have a Python virtual environment set up, it would also be beneficial to have access to a GPU compatible with CUDA for acceleration.\npython -m venv venv source venv/bin/activate To utilize Pyannote.audio models for diarization through WhisperX, you will need a Hugging Face access token with \u0026lsquo;read\u0026rsquo; permissions, for which you must accept the user agreements for the pyannote/segmentation-3.0 and pyannote/speaker-diarization-3.1 models on the Hugging Face website: HuggingFace\nFinally we can pip install our dependencies:\npython -m pip install torch numpy pyannote.audio whisper whisperx First, we\u0026rsquo;ll import the necessary libraries, and configure our Hugging Face token and file path.\nimport torch import whisper from pyannote.audio import Pipeline from whisperx.diarize import DiarizationPipeline,assign_word_speakers from whisperx import load_align_model, align hugging_face_token=\u0026#34;hf_token\u0026#34; audio_path=\u0026#34;audio_file_path\u0026#34; # Check if cuda is available import torch DEVICE = torch.device(\u0026#34;cuda\u0026#34; if torch.cuda.is_available() else \u0026#34;cpu\u0026#34;) DEVICE Diarization In the first code snippet, a pre-trained pyannote model is utilized to determine when different speakers are active in an audio recording. The output of this speaker segmentation can be viewed via the print statement:\n#Initialize pipeline pipeline = Pipeline.from_pretrained( \u0026#34;pyannote/speaker-diarization-3.1\u0026#34;, use_auth_token=hugging_face_token) # send pipeline to GPU (when available) pipeline.to(torch.device(\u0026#34;cuda\u0026#34;)) # apply pretrained pipeline diarization = pipeline(audio_path) # print the result for turn, _, speaker in diarization.itertracks(yield_label=True): print(f\u0026#34;start={turn.start:.1f}s stop={turn.end:.1f}s speaker_{speaker}\u0026#34;) This plot illustrates the various segments identified by the model throughout the podcast\u0026rsquo;s duration.\nTranscription Now let\u0026rsquo;s run the Whisper model to generate the transcription:\nmodel_name = \u0026#34;turbo\u0026#34; model = whisper.load_model(model_name, DEVICE) script = model.transcribe(audio_path) Naive alignment The following script attempts a basic diarization-to-transcription alignment. It directly links diarization segments (which identify who spoke when) with Whisper\u0026rsquo;s transcription segments (which capture what was said) by simply checking if their timestamps overlap. However, this straightforward timestamp-based matching is often insufficient and can lead to significant inaccuracies, such as misaligned text, missing words, or incorrect speaker attribution, because it doesn\u0026rsquo;t account for precise word timings or subtle overlaps in speech.\n# Create a text associating diarization segments with Whisper transcription output_text = [] for turn, _, speaker in diarization.itertracks(yield_label=True): segment_text = [] for segment in script[\u0026#34;segments\u0026#34;]: if segment[\u0026#34;start\u0026#34;] \u0026gt;= turn.start and segment[\u0026#34;end\u0026#34;] \u0026lt;= turn.end: segment_text.append(segment[\u0026#34;text\u0026#34;]) combined_text = \u0026#34; \u0026#34;.join(segment_text) output_text.append(f\u0026#34;{speaker} [{turn.start:.2f}s - {turn.end:.2f}s]: {combined_text}\u0026#34;) # Print the result for line in output_text: print(line) Advanced alignment - WhisperX While a simple timestamp overlap can lead to significant inaccuracies in aligning diarization and transcription segments, WhisperX addresses these challenges through forced alignment. Unlike basic timestamp matching, WhisperX refines the alignment by first leveraging the more precise word-level timestamps generated by its underlying Whisper model. It then performs a forced alignment process, which essentially \u0026ldquo;snaps\u0026rdquo; the diarization segments to the exact start and end times of the individual words within the transcription. This method accounts for subtle overlaps and ensures that each word is attributed to the correct speaker, even when speech segments are close together or overlap slightly.\nThis script first uses the pyannote library to identify speakers and time of enunciation in an audio file (that\u0026rsquo;s the \u0026ldquo;diarization\u0026rdquo; part). Then, it takes a pre-generated text transcript (script[\u0026ldquo;segments\u0026rdquo;]) and uses WhisperX\u0026rsquo;s alignment models to precisely match each word in the transcript to its exact timing in the audio. Finally, it combines the speaker information from pyannote with the word-level timings from WhisperX to produce a highly accurate, time-stamped transcript where each spoken word is correctly attributed to the right speaker.\n# Initialize a diarization pipeline diarization_pipeline = DiarizationPipeline(use_auth_token=hugging_face_token) diarized = diarization_pipeline(audio_path) # Align Script model_a, metadata = load_align_model(language_code=script[\u0026#34;language\u0026#34;], device=DEVICE) script_aligned = align(script[\u0026#34;segments\u0026#34;], model_a, metadata, audio_path, DEVICE) # Align Speakers result_segments, word_seg = list(assign_word_speakers( diarized, script_aligned ).values()) transcribed = [] for result_segment in result_segments: transcribed.append( { \u0026#34;start\u0026#34;: result_segment[\u0026#34;start\u0026#34;], \u0026#34;end\u0026#34;: result_segment[\u0026#34;end\u0026#34;], \u0026#34;text\u0026#34;: result_segment[\u0026#34;text\u0026#34;], \u0026#34;speaker\u0026#34;: result_segment[\u0026#34;speaker\u0026#34;], } ) for start, end, text, speaker in [i.values() for i in transcribed]: print(start, end, speaker, text) By combining Pyannote\u0026rsquo;s precise speaker diarization with WhisperX\u0026rsquo;s advanced forced alignment, this guide demonstrates how to generate highly accurate, speaker-attributed transcripts. This approach significantly surpasses basic timestamp matching, ensuring precise information about \u0026ldquo;who spoke when and what was said\u0026rdquo;.\nReferences : PyAnnote Whisper WhisperX ","content_html":"\u003cp\u003eSpeaker Diarization answers the question \u0026ldquo;who spoke when?\u0026rdquo; by segmenting an audio stream based on speaker identity, while transcription tells us \u0026ldquo;what was said\u0026rdquo;.\u003c/p\u003e\n\u003ch3 id=\"introducing-the-key-components\"\u003eIntroducing the Key Components\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eWhisper\u003c/strong\u003e: Automatic Speech Recognition model that converts speech to text.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003ePyannote.audio\u003c/strong\u003e: Deep neural network for speaker diarization.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWhisperX\u003c/strong\u003e: Optimized Whisper variant that integrates multiple components.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"a-step-by-step-implementation-guide\"\u003eA Step-by-Step Implementation Guide\u003c/h3\u003e\n\u003cp\u003eThis section provides a practical guide to implementing speaker diarization using WhisperX, Pyannote, and Whisper, while highlighting WhisperX\u0026rsquo;s advantages. For our audio sample, we can grab a segment from pretty much any free podcast out there – let\u0026rsquo;s go for a clip from \u003ca href=\"https://lexfridman.com/podcast/\"\u003eLex Friedman\u003c/a\u003e, one of my personal favorites.\u003c/p\u003e\n\u003ch4 id=\"setup\"\u003eSetup\u003c/h4\u003e\n\u003cp\u003eBefore diving into the code, it is essential to prepare your development environment.\nMake sure you have a Python virtual environment set up, it would also be beneficial to have access to a GPU compatible with CUDA for acceleration.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003epython -m venv venv\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003esource\u003c/span\u003e venv/bin/activate\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eTo utilize Pyannote.audio models for diarization through WhisperX, you will need a Hugging Face access token with \u0026lsquo;read\u0026rsquo; permissions, for which you must accept the user agreements for the pyannote/segmentation-3.0 and pyannote/speaker-diarization-3.1 models on the Hugging Face website: \u003ca href=\"https://github.com/pyannote/pyannote-audio\"\u003eHuggingFace\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eFinally we can pip install our dependencies:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003epython -m pip  install  torch  numpy pyannote.audio  whisper  whisperx\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eFirst, we\u0026rsquo;ll import the necessary libraries, and configure our Hugging Face token and file path.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etorch\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ewhisper\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003epyannote.audio\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ePipeline\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ewhisperx.diarize\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eDiarizationPipeline\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eassign_word_speakers\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ewhisperx\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eload_align_model\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ealign\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003ehugging_face_token\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;hf_token\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eaudio_path\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;audio_file_path\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# Check if cuda is available\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etorch\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eDEVICE\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etorch\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edevice\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;cuda\u0026#34;\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etorch\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecuda\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eis_available\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003eelse\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;cpu\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eDEVICE\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch4 id=\"diarization\"\u003eDiarization\u003c/h4\u003e\n\u003cp\u003eIn the first code snippet, a pre-trained pyannote model is utilized to determine when different speakers are active in an audio recording. The output of this speaker segmentation can be viewed via the print statement:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e#Initialize pipeline\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003epipeline\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ePipeline\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003efrom_pretrained\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#d88200\"\u003e\u0026#34;pyannote/speaker-diarization-3.1\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003euse_auth_token\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ehugging_face_token\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# send pipeline to GPU (when available)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003epipeline\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eto\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etorch\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003edevice\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;cuda\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# apply pretrained pipeline\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003ediarization\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003epipeline\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eaudio_path\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# print the result\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eturn\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e_\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003espeaker\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ediarization\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eitertracks\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eyield_label\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eTrue\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;start=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eturn\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estart\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e:\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e.1f\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003es stop=\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eturn\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eend\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e:\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e.1f\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003es speaker_\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003espeaker\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis plot illustrates the various segments identified by the model throughout the podcast\u0026rsquo;s duration.\u003c/p\u003e\n\u003cfigure\u003e\n  \u003cimg src=\"../../images/diarization_output.png\" alt=\"Diarization\"\u003e\n\u003c/figure\u003e\n\u003ch4 id=\"transcription\"\u003eTranscription\u003c/h4\u003e\n\u003cp\u003eNow let\u0026rsquo;s run the Whisper model to generate the transcription:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003emodel_name\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34;turbo\u0026#34;\u003c/span\u003e  \n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ewhisper\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eload_model\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003emodel_name\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eDEVICE\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003escript\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emodel\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003etranscribe\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eaudio_path\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch4 id=\"naive-alignment\"\u003eNaive alignment\u003c/h4\u003e\n\u003cp\u003eThe following script attempts a basic diarization-to-transcription alignment. It directly links diarization segments (which identify who spoke when) with Whisper\u0026rsquo;s transcription segments (which capture what was said) by simply checking if their timestamps overlap. However, this straightforward timestamp-based matching is often insufficient and can lead to significant inaccuracies, such as misaligned text, missing words, or incorrect speaker attribution, because it doesn\u0026rsquo;t account for precise word timings or subtle overlaps in speech.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# Create a text associating diarization segments with Whisper transcription\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eoutput_text\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eturn\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e_\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003espeaker\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ediarization\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eitertracks\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eyield_label\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#00a8c8\"\u003eTrue\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003esegment_text\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esegment\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003escript\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;segments\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#00a8c8\"\u003eif\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esegment\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;start\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e\u0026gt;=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eturn\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estart\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eand\u003c/span\u003e \u003cspan style=\"color:#111\"\u003esegment\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;end\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e\u0026lt;=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eturn\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eend\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#111\"\u003esegment_text\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eappend\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esegment\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;text\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e])\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ecombined_text\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#d88200\"\u003e\u0026#34; \u0026#34;\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ejoin\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003esegment_text\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eoutput_text\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eappend\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003ef\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003espeaker\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e [\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eturn\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estart\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e:\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e.2f\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003es - \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eturn\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eend\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e:\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e.2f\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003es]: \u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e{\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ecombined_text\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e}\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# Print the result\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eline\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eoutput_text\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eline\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch4 id=\"advanced-alignment---whisperx\"\u003eAdvanced alignment - WhisperX\u003c/h4\u003e\n\u003cp\u003eWhile a simple timestamp overlap can lead to significant inaccuracies in aligning diarization and transcription segments, WhisperX addresses these challenges through forced alignment. Unlike basic timestamp matching, WhisperX refines the alignment by first leveraging the more precise word-level timestamps generated by its underlying Whisper model. It then performs a forced alignment process, which essentially \u0026ldquo;snaps\u0026rdquo; the diarization segments to the exact start and end times of the individual words within the transcription. This method accounts for subtle overlaps and ensures that each word is attributed to the correct speaker, even when speech segments are close together or overlap slightly.\u003c/p\u003e\n\u003cfigure\u003e\n  \u003cimg src=\"../../images/whisperx.png\" alt=\"whisperx\"\u003e\n\u003c/figure\u003e\n\u003cp\u003eThis script first uses the pyannote library to identify speakers and time of enunciation in an audio file (that\u0026rsquo;s the \u0026ldquo;diarization\u0026rdquo; part). Then, it takes a pre-generated text transcript (script[\u0026ldquo;segments\u0026rdquo;]) and uses WhisperX\u0026rsquo;s alignment models to precisely match each word in the transcript to its exact timing in the audio. Finally, it combines the speaker information from pyannote with the word-level timings from WhisperX to produce a highly accurate, time-stamped transcript where each spoken word is correctly attributed to the right speaker.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#272822;background-color:#fafafa;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# Initialize a diarization pipeline\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003ediarization_pipeline\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eDiarizationPipeline\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003euse_auth_token\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ehugging_face_token\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003ediarized\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ediarization_pipeline\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eaudio_path\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# Align Script\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003emodel_a\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emetadata\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eload_align_model\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003elanguage_code\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003escript\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;language\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003edevice\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eDEVICE\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003escript_aligned\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ealign\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003escript\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;segments\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emodel_a\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003emetadata\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eaudio_path\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eDEVICE\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# Align Speakers\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003eresult_segments\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eword_seg\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003elist\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eassign_word_speakers\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003ediarized\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003escript_aligned\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003evalues\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#111\"\u003etranscribed\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresult_segment\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresult_segments\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003etranscribed\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003eappend\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;start\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresult_segment\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;start\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;end\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresult_segment\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;end\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;text\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresult_segment\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;text\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e            \u003cspan style=\"color:#d88200\"\u003e\u0026#34;speaker\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e:\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eresult_segment\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#d88200\"\u003e\u0026#34;speaker\u0026#34;\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e],\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e        \u003cspan style=\"color:#111\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003estart\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eend\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etext\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003espeaker\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003e[\u003c/span\u003e\u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\u003cspan style=\"color:#111\"\u003evalues\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e()\u003c/span\u003e \u003cspan style=\"color:#00a8c8\"\u003efor\u003c/span\u003e \u003cspan style=\"color:#111\"\u003ei\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etranscribed\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e]:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#111\"\u003eprint\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e(\u003c/span\u003e\u003cspan style=\"color:#111\"\u003estart\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003eend\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003espeaker\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e,\u003c/span\u003e \u003cspan style=\"color:#111\"\u003etext\u003c/span\u003e\u003cspan style=\"color:#111\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eBy combining Pyannote\u0026rsquo;s precise speaker diarization with WhisperX\u0026rsquo;s advanced forced alignment, this guide demonstrates how to generate highly accurate, speaker-attributed transcripts. This approach significantly surpasses basic timestamp matching, ensuring precise information about \u0026ldquo;who spoke when and what was said\u0026rdquo;.\u003c/p\u003e\n\u003ch4 id=\"references-\"\u003eReferences :\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/pyannote/pyannote-audio\"\u003ePyAnnote\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/openai/whisper\"\u003eWhisper\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/m-bain/whisperX\"\u003eWhisperX\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n","url":"http://gabelfra1.github.io/gabelfra.github.io/posts/asr/","date_published":"6066-06-09T60:66:00+00:00","date_modified":"6066-06-09T60:66:00+00:00","author":{"name":"Calvin Tran","url":"http://gabelfra1.github.io/gabelfra.github.io/"}}]}