Tutorial

This tutorial walks through a complete Cobalt workflow: training a model on synthetic data, preparing the data for Cobalt, finding failure groups, and launching the interactive UI.

Generate Synthetic Data

First, generate a synthetic classification dataset and split it into training and test sets:

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=5000, random_state=73902)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=73902
)

Train a Model

Train a random forest classifier on the training data:

from sklearn.ensemble import RandomForestClassifier

rf_model = RandomForestClassifier(random_state=73902)
rf_model.fit(X_train, y_train)

Prepare a DataFrame

Create a Pandas DataFrame containing the test features, ground truth labels, and model predictions:

import pandas as pd

y_pred = rf_model.predict(X_test)
df = pd.DataFrame(X_test, columns=[f"feat_{i}" for i in range(20)])
df["y_true"] = y_test
df["y_pred"] = y_pred

Create a CobaltDataset

Create a CobaltDataset from the DataFrame:

from cobalt import CobaltDataset

ds = CobaltDataset(df)

Add Model Information

Tell Cobalt which columns contain the model inputs, ground truth, and predictions:

ds.add_model(
    input_columns=[f"feat_{i}" for i in range(20)],
    target_column="y_true",
    prediction_column="y_pred",
    task="classification",
    name="rf",
)

Compute Model Performance Metrics

Compute standard performance metrics for the model:

ds.compute_model_performance_metrics()

Extract Embeddings from the Random Forest

Use the random forest’s leaf node assignments as embeddings. Each data point is mapped to the set of leaf nodes it lands in across all trees, producing a high-dimensional binary vector that captures the model’s learned similarity structure:

import numpy as np

leaf_nodes = rf_model.apply(X_test)
ds.add_embedding_array(leaf_nodes, metric="hamming", name="rf")

Create a DatasetSplit and Workspace

Wrap the dataset in a DatasetSplit and create a Workspace for analysis:

from cobalt import DatasetSplit, Workspace

split = DatasetSplit(ds, {"test": np.arange(len(X_test))})
w = Workspace(split)

Find Failure Groups

Run Cobalt’s failure group detection to find clusters of data points where the model underperforms:

w.find_failure_groups()

This returns a table of failure groups ranked by severity. Each group represents a semantically coherent cluster of data points with elevated error rates compared to the overall model performance.

Group	Size	Error Rate	Description
0	42	0.45	High feat_3, low feat_7
1	38	0.39	Cluster near decision boundary
2	27	0.37	Outlier region in feat_12

Launch the UI

Open the interactive Cobalt UI to explore the results visually:

w.ui()

This launches an interactive visualization in your notebook where you can explore the network map of your dataset, inspect failure groups, and drill down into individual data points.