Semantic segmentation with a Tsetlin machine¶

A convolutional Tsetlin machine ORs a clause over every patch of an image: it can say that a pattern occurs, but not where. That is fine for classifying an image and useless for labelling its pixels. SegmentationTsetlinMachine keeps the patch axis instead. Every pixel is one example, classified from its k × k neighbourhood, and the prediction is a label map.

This notebook follows the documentation's segmentation example (the script examples/segmentation_scenes.py) step by step:

1. generated CamVid-like street scenes, and booleanizing colour with a thermometer;
2. the floor every result should be reported against: a baseline that never looks at a pixel;
3. a single-scale model trained with the Trainer;
4. adding context the 3 × 3 window does not contain: position literals and an OR-pooled pyramid;
5. per-class metrics and boundary F1;
6. why a pixel got its label: clauses as statements about named pixels;
7. patches_per_commit, the fidelity knob of dense feedback;
8. class-balanced feedback.

The segmentation guide covers the API and Segmentation the algorithm. Notebook 05 uses the same scenes to review the CTM-UNet paper.

Written for a GPU (about 3 minutes on an RTX 3090) and needs no download. In Colab choose Runtime → Change runtime type → T4 GPU; expect it to take longer there. On a CPU it runs, but much more slowly.

In [ ]:

# Setup: on Google Colab this installs torchtsetlin from PyPI (torch, torchvision, scikit-learn,
# matplotlib and pandas are already there). Elsewhere: `pip install torchtsetlin`, or
# `pip install -e ".[dev]"` in a checkout of the repository.
import sys

IN_COLAB = "google.colab" in sys.modules
if IN_COLAB:
    %pip install -q "torchtsetlin>=0.7"

# Setup: on Google Colab this installs torchtsetlin from PyPI (torch, torchvision, scikit-learn, # matplotlib and pandas are already there). Elsewhere: `pip install torchtsetlin`, or # `pip install -e ".[dev]"` in a checkout of the repository. import sys IN_COLAB = "google.colab" in sys.modules if IN_COLAB: %pip install -q "torchtsetlin>=0.7"

In [ ]:

import time

import matplotlib.pyplot as plt
import pandas as pd
import torch

import torchtsetlin as tt

SEED = 0
tt.seed_everything(SEED)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
CLASSES = list(tt.data.SCENE_CLASSES)
print("torchtsetlin", tt.__version__, "| torch", torch.__version__, "| device", device)
if device.type == "cuda":
    print("gpu:", torch.cuda.get_device_name(0))

import time import matplotlib.pyplot as plt import pandas as pd import torch import torchtsetlin as tt SEED = 0 tt.seed_everything(SEED) device = torch.device("cuda" if torch.cuda.is_available() else "cpu") CLASSES = list(tt.data.SCENE_CLASSES) print("torchtsetlin", tt.__version__, "| torch", torch.__version__, "| device", device) if device.type == "cuda": print("gpu:", torch.cuda.get_device_name(0))

1. The scenes¶

tt.data.make_scenes generates street scenes: a wavy horizon, a large sky, buildings standing on the horizon, trees and a road. Each class has its own texture (window grids, foliage noise, lane markings) and every scene a random illumination, so colour alone does not separate the classes. The class frequencies are as unbalanced as in CamVid.

In [ ]:

N_TRAIN, N_TEST, SIZE = 220, 80, 32
rgb, labels = tt.data.make_scenes(N_TRAIN + N_TEST, SIZE, seed=SEED, device=device)
print("images", tuple(rgb.shape), rgb.dtype, "| labels", tuple(labels.shape), labels.dtype)

freq = torch.bincount(labels.reshape(-1), minlength=len(CLASSES)).float()
print("class frequency:", "  ".join(f"{c}={p:.3f}" for c, p in zip(CLASSES, (freq / freq.sum()).tolist())))

for i in range(3):
    tt.viz.plot_segmentation(rgb[i], labels[i], class_names=CLASSES, n_classes=len(CLASSES),
                             titles=["scene", "labels"], figsize_per=2.5)
    plt.show()

N_TRAIN, N_TEST, SIZE = 220, 80, 32 rgb, labels = tt.data.make_scenes(N_TRAIN + N_TEST, SIZE, seed=SEED, device=device) print("images", tuple(rgb.shape), rgb.dtype, "| labels", tuple(labels.shape), labels.dtype) freq = torch.bincount(labels.reshape(-1), minlength=len(CLASSES)).float() print("class frequency:", " ".join(f"{c}={p:.3f}" for c, p in zip(CLASSES, (freq / freq.sum()).tolist()))) for i in range(3): tt.viz.plot_segmentation(rgb[i], labels[i], class_names=CLASSES, n_classes=len(CLASSES), titles=["scene", "labels"], figsize_per=2.5) plt.show()

Booleanizing colour¶

A Tsetlin machine reads Boolean literals. A single threshold per channel throws away the ordering between intensities, which is most of what a segmenter needs. ColorThermometerEncoder gives each channel n_bits planes, plane k being "value ≥ k-th level". With 4 bits, the 3 colour channels become 12 Boolean planes: 0–3 red, 4–7 green, 8–11 blue.

In [ ]:

enc = tt.data.ColorThermometerEncoder(n_bits=4, value_range=(0.0, 1.0)).to(device)
x = enc(rgb).float()
print("Boolean planes", tuple(x.shape), "| fraction ON", round(x.mean().item(), 3))

x_tr, y_tr = x[:N_TRAIN], labels[:N_TRAIN]
x_te, y_te = x[N_TRAIN:], labels[N_TRAIN:]
rgb_te = rgb[N_TRAIN:]

fig, axes = plt.subplots(1, 4, figsize=(10, 2.6))
for k, ax in enumerate(axes):
    ax.imshow(x[0, k].cpu(), cmap="gray", interpolation="nearest")
    ax.set_title(f"red plane {k}")
    ax.axis("off")
plt.tight_layout()
plt.show()

enc = tt.data.ColorThermometerEncoder(n_bits=4, value_range=(0.0, 1.0)).to(device) x = enc(rgb).float() print("Boolean planes", tuple(x.shape), "| fraction ON", round(x.mean().item(), 3)) x_tr, y_tr = x[:N_TRAIN], labels[:N_TRAIN] x_te, y_te = x[N_TRAIN:], labels[N_TRAIN:] rgb_te = rgb[N_TRAIN:] fig, axes = plt.subplots(1, 4, figsize=(10, 2.6)) for k, ax in enumerate(axes): ax.imshow(x[0, k].cpu(), cmap="gray", interpolation="nearest") ax.set_title(f"red plane {k}") ax.axis("off") plt.tight_layout() plt.show()

2. The floor: a baseline that never looks at a pixel¶

These scenes have a fixed layout: sky on top, road at the bottom. A "classifier" that labels every pixel with the most common class of its row in the training set needs no image at all. Its score is the floor any segmentation result on this data has to be read against.

In [ ]:

def row_prior(l_train, l_test):
    H = l_train.shape[1]
    mode = torch.stack([torch.bincount(l_train[:, r, :].reshape(-1), minlength=len(CLASSES)).argmax()
                        for r in range(H)])
    return mode.view(1, H, 1).expand_as(l_test)


baseline = tt.metrics.segmentation_metrics(row_prior(y_tr, y_te), y_te, len(CLASSES), CLASSES)
print(f"row prior: pixel accuracy {baseline['pixel_accuracy']:.3f}   mean IoU {baseline['mean_iou']:.3f}")
print("           " + "  ".join(f"{c} {baseline['iou_' + c]:.3f}" for c in CLASSES))

def row_prior(l_train, l_test): H = l_train.shape[1] mode = torch.stack([torch.bincount(l_train[:, r, :].reshape(-1), minlength=len(CLASSES)).argmax() for r in range(H)]) return mode.view(1, H, 1).expand_as(l_test) baseline = tt.metrics.segmentation_metrics(row_prior(y_tr, y_te), y_te, len(CLASSES), CLASSES) print(f"row prior: pixel accuracy {baseline['pixel_accuracy']:.3f} mean IoU {baseline['mean_iou']:.3f}") print(" " + " ".join(f"{c} {baseline['iou_' + c]:.3f}" for c in CLASSES))

Pixel accuracy is high because sky and road are most of every image; mean IoU, which weighs every class equally, shows that the baseline never finds a building or a tree. Read mean IoU.

3. A single-scale model¶

SegmentationTsetlinMachine(n_classes, n_clauses, ...) with:

• patch_size=3: each pixel is classified from its 3 × 3 neighbourhood, 12 × 9 = 108 features;
• n_clauses=300 clauses per class, T=60, s=10, weighted clauses with 100 states per automaton;
• patches_per_commit=128 (the default): see section 7.

The Trainer recognises a dense model and reports segmentation metrics. batch_size counts images: a batch of 8 is 8 192 pixel-examples.

In [ ]:

RESULTS = {"row prior (no image content)": baseline}


def train(name, xt, xv, patch_size=3, position=False, class_p=None, patches_per_commit=128, epochs=8):
    tt.seed_everything(SEED)
    model = tt.SegmentationTsetlinMachine(
        len(CLASSES), n_clauses=300, T=60, s=10.0, patch_size=patch_size,
        position_encoding=position, patches_per_commit=patches_per_commit,
        weighted=True, n_states=100,
    ).to(device)
    model.class_names = CLASSES
    if class_p is not None:
        model.set_class_feedback_p(class_p)
    t0 = time.time()
    trainer = tt.Trainer(model, device=device, batch_size=8, verbose=False)
    trainer.fit((xt, y_tr), epochs=epochs)
    res = trainer.test((xv, y_te))
    seconds = time.time() - t0
    model.eval()                     # empty clauses are False when predicting: never skip this
    res = {k.replace("test_", ""): v for k, v in res.items()}
    res["boundary_f1"] = tt.metrics.boundary_f1(model.predict(xv), y_te, tolerance=2)
    res["seconds"] = seconds
    RESULTS[name] = res
    print(f"{name:<24} pixel acc {res['pixel_accuracy']:.3f}  mIoU {res['mean_iou']:.3f}  "
          f"building IoU {res['iou_building']:.3f}  [{seconds:.0f} s]")
    return model


single = train("single scale 3x3", x_tr, x_te)

RESULTS = {"row prior (no image content)": baseline} def train(name, xt, xv, patch_size=3, position=False, class_p=None, patches_per_commit=128, epochs=8): tt.seed_everything(SEED) model = tt.SegmentationTsetlinMachine( len(CLASSES), n_clauses=300, T=60, s=10.0, patch_size=patch_size, position_encoding=position, patches_per_commit=patches_per_commit, weighted=True, n_states=100, ).to(device) model.class_names = CLASSES if class_p is not None: model.set_class_feedback_p(class_p) t0 = time.time() trainer = tt.Trainer(model, device=device, batch_size=8, verbose=False) trainer.fit((xt, y_tr), epochs=epochs) res = trainer.test((xv, y_te)) seconds = time.time() - t0 model.eval() # empty clauses are False when predicting: never skip this res = {k.replace("test_", ""): v for k, v in res.items()} res["boundary_f1"] = tt.metrics.boundary_f1(model.predict(xv), y_te, tolerance=2) res["seconds"] = seconds RESULTS[name] = res print(f"{name:<24} pixel acc {res['pixel_accuracy']:.3f} mIoU {res['mean_iou']:.3f} " f"building IoU {res['iou_building']:.3f} [{seconds:.0f} s]") return model single = train("single scale 3x3", x_tr, x_te)

In [ ]:

def show(model, xv, i=0, title=None):
    pred = model.predict(xv[i:i + 1])[0]
    tt.viz.plot_segmentation(rgb_te[i], y_te[i], pred, class_names=CLASSES, n_classes=len(CLASSES),
                             figsize_per=2.5)
    if title:
        plt.suptitle(title, y=1.08)
    plt.show()


show(single, x_te, title="single scale 3x3")

def show(model, xv, i=0, title=None): pred = model.predict(xv[i:i + 1])[0] tt.viz.plot_segmentation(rgb_te[i], y_te[i], pred, class_names=CLASSES, n_classes=len(CLASSES), figsize_per=2.5) if title: plt.suptitle(title, y=1.08) plt.show() show(single, x_te, title="single scale 3x3")

In our run the single-scale model reaches about 0.945 pixel accuracy and 0.82 mean IoU, against 0.876 and 0.410 for the row prior. Sky and road are almost solved. Building is the weak class (IoU about 0.53): its pixels are confused with tree and sky along the band above the horizon, which is where the errors in the figure sit.

4. Context the window does not contain¶

A 3 × 3 patch of building interior and a 3 × 3 patch of road look alike once illumination varies. What separates them is where they are, relative to the horizon, and that is outside the window. Enlarging the patch grows the feature count quadratically. Two cheaper ways to add context stay within Boolean logic:

• Position literals (position_encoding=8): each pixel also gets thermometer-coded row and column bits, so a clause can say "only below row 12". Here 8 bits per axis.
• An OR-pooled pyramid (tt.data.PyramidEncoder(scales=(1, 2, 4), patch=3)): the planes are downsampled by OR-pooling to 1/2 and 1/4 resolution, and each pixel gets the 3 × 3 patch at every level as extra planes. The model then uses patch_size=1, because the neighbourhood is already in the features: 12 × 9 × 3 = 324 planes.

In [ ]:

pyr = tt.data.PyramidEncoder(scales=(1, 2, 4), patch=3)
px_tr, px_te = pyr(x_tr).float(), pyr(x_te).float()
print("pyramid planes", tuple(px_tr.shape))

position = train("+ position literals", x_tr, x_te, position=8)
pyramid = train("pyramid 1x/2x/4x", px_tr, px_te, patch_size=1)
both = train("pyramid + position", px_tr, px_te, patch_size=1, position=8)

pyr = tt.data.PyramidEncoder(scales=(1, 2, 4), patch=3) px_tr, px_te = pyr(x_tr).float(), pyr(x_te).float() print("pyramid planes", tuple(px_tr.shape)) position = train("+ position literals", x_tr, x_te, position=8) pyramid = train("pyramid 1x/2x/4x", px_tr, px_te, patch_size=1) both = train("pyramid + position", px_tr, px_te, patch_size=1, position=8)

In [ ]:

cols = ["pixel_accuracy", "mean_iou"] + [f"iou_{c}" for c in CLASSES] + ["boundary_f1", "seconds"]
table = pd.DataFrame(RESULTS).T.reindex(columns=cols)
table.round(3)

cols = ["pixel_accuracy", "mean_iou"] + [f"iou_{c}" for c in CLASSES] + ["boundary_f1", "seconds"] table = pd.DataFrame(RESULTS).T.reindex(columns=cols) table.round(3)

In our run both kinds of context help, and they compose:

• Position literals lift building IoU from about 0.53 to about 0.87, the pyramid to about 0.84, and the two together to about 0.92 (mean IoU about 0.95, pixel accuracy about 0.993).
• Boundary F1 rises from about 0.77 to about 0.98: the outlines, not only the regions, come right.
• None of it costs time here: every model trains in about the same 16–19 s.

The building column is the one to read. The window could not tell building from its neighbours; knowing the row (position), or seeing a coarser picture of the surroundings (pyramid), can.

In [ ]:

for model, xv, name in [(single, x_te, "single scale 3x3"), (both, px_te, "pyramid + position")]:
    show(model, xv, i=3, title=name)

for model, xv, name in [(single, x_te, "single scale 3x3"), (both, px_te, "pyramid + position")]: show(model, xv, i=3, title=name)

5. Per-class metrics¶

tt.metrics.segmentation_report prints IoU, Dice, precision and recall per class from a label map. Boundary F1 (in the table above) scores only the class edges, within a tolerance of 2 pixels: it is the metric that shows whether the outlines are right, which mean IoU over large regions hides.

In [ ]:

print(tt.metrics.segmentation_report(both.predict(px_te), y_te, class_names=CLASSES))

print(tt.metrics.segmentation_report(both.predict(px_te), y_te, class_names=CLASSES))

The vote map shows the model's confidence surface for one class. Red is for "building", blue against; the boundary sits where the margin collapses.

In [ ]:

fig, axes = plt.subplots(1, 2, figsize=(8, 3.5))
axes[0].imshow(rgb_te[3].permute(1, 2, 0).cpu())
axes[0].set_title("scene")
axes[0].axis("off")
tt.viz.plot_vote_map(both.vote_map(px_te[3:4]), class_index=CLASSES.index("building"),
                     class_name="building", ax=axes[1])
plt.tight_layout()
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(8, 3.5)) axes[0].imshow(rgb_te[3].permute(1, 2, 0).cpu()) axes[0].set_title("scene") axes[0].axis("off") tt.viz.plot_vote_map(both.vote_map(px_te[3:4]), class_index=CLASSES.index("building"), class_name="building", ax=axes[1]) plt.tight_layout() plt.show()

6. Why did a pixel get its label?¶

Each clause is an AND of literals over the pixel's neighbourhood. Read at a given pixel, tt.interpret.explain_pixel names the literals by absolute image coordinates: p[5,20,12] is "plane 5 (green, level 2) of pixel (20, 12) is on". Position literals print as y>k / x>k. The single-scale model with position literals has the most readable clauses (12 planes rather than 324), so we ask it about a road pixel.

In [ ]:

ROW, COL = 20, 12
expl = tt.interpret.explain_pixel(position, x_te[0], ROW, COL, target=int(y_te[0, ROW, COL]), max_clauses=4)
print("true label:", CLASSES[int(y_te[0, ROW, COL])])
print(expl)
print()
for m in expl.matched:          # position literals decoded to the rows and columns a clause allows
    print(f"clause {m.clause}: {position.clause_region(m.clause)}")

ROW, COL = 20, 12 expl = tt.interpret.explain_pixel(position, x_te[0], ROW, COL, target=int(y_te[0, ROW, COL]), max_clauses=4) print("true label:", CLASSES[int(y_te[0, ROW, COL])]) print(expl) print() for m in expl.matched: # position literals decoded to the rows and columns a clause allows print(f"clause {m.clause}: {position.clause_region(m.clause)}")

Planes 0–3 are red at levels ≥ 0.2, 0.4, 0.6, 0.8; 4–7 green and 8–11 blue likewise. y>k is a row threshold, decoded by clause_region in the last lines of the output.

In our run the strongest votes for "road" come from clauses that say only "the row is 18 or lower" (y>2 AND y>3 AND y>4): the layout prior that the row-prior baseline used, learned as single clauses. The next one is a colour statement about the 3 × 3 neighbourhood, restricted to rows 11–20: some red and blue (p[0,…], p[8,…]) but none of the brighter green and blue levels (NOT p[5,…], NOT p[6,…], NOT p[10,…]), i.e. a dim, grey pixel, which is what asphalt looks like in these scenes. Each clause can be read and checked against the image.

model.clause_map(x, clauses) shows where clauses fire over a whole image, and model.clause_region(j) decodes a clause's position literals into the rows and columns it is pinned to. Below, for each class, the clause that adds the most votes to that class's pixels of one test scene, and where it fires:

In [ ]:

# Clauses are owned by classes, 300 each: the first 150 vote for the class, the last 150 against.
# For each class, take the clause that adds the most votes to that class's own pixels of one scene.
SCENE = 0
fires = position.clause_map(x_te[SCENE:SCENE + 1])[0]                  # (C, H, W) bool
picked = {}
for k, name in enumerate(CLASSES):
    ids = torch.arange(k * 300, k * 300 + 150, device=fires.device)
    on_class = (fires[ids] & (y_te[SCENE] == k)).flatten(1).sum(1)     # pixels of class k it fires on
    score = on_class * position.weights[ids]
    j = int(ids[score.argmax()])
    picked[name] = j
    region = position.clause_region(j)
    print(f"{name:<9} clause {j:>4}  weight {int(position.weights[j]):>4}  "
          f"fires on {int(fires[j].sum()):>3} px  rows {region['y']}  cols {region['x']}")

maps = fires[list(picked.values())].cpu()
fig, axes = plt.subplots(1, len(CLASSES) + 1, figsize=(13, 2.8))
axes[0].imshow(rgb_te[SCENE].permute(1, 2, 0).cpu())
axes[0].set_title("scene")
for ax, (name, j), m in zip(axes[1:], picked.items(), maps):
    ax.imshow(m, cmap="gray", vmin=0, vmax=1, interpolation="nearest")
    ax.set_title(f"{name}: clause {j}")
for ax in axes:
    ax.set_xticks([])
    ax.set_yticks([])
plt.tight_layout()
plt.show()

# Clauses are owned by classes, 300 each: the first 150 vote for the class, the last 150 against. # For each class, take the clause that adds the most votes to that class's own pixels of one scene. SCENE = 0 fires = position.clause_map(x_te[SCENE:SCENE + 1])[0] # (C, H, W) bool picked = {} for k, name in enumerate(CLASSES): ids = torch.arange(k * 300, k * 300 + 150, device=fires.device) on_class = (fires[ids] & (y_te[SCENE] == k)).flatten(1).sum(1) # pixels of class k it fires on score = on_class * position.weights[ids] j = int(ids[score.argmax()]) picked[name] = j region = position.clause_region(j) print(f"{name:<9} clause {j:>4} weight {int(position.weights[j]):>4} " f"fires on {int(fires[j].sum()):>3} px rows {region['y']} cols {region['x']}") maps = fires[list(picked.values())].cpu() fig, axes = plt.subplots(1, len(CLASSES) + 1, figsize=(13, 2.8)) axes[0].imshow(rgb_te[SCENE].permute(1, 2, 0).cpu()) axes[0].set_title("scene") for ax, (name, j), m in zip(axes[1:], picked.items(), maps): ax.imshow(m, cmap="gray", vmin=0, vmax=1, interpolation="nearest") ax.set_title(f"{name}: clause {j}") for ax in axes: ax.set_xticks([]) ax.set_yticks([]) plt.tight_layout() plt.show()

7. patches_per_commit: how much feedback is applied at once¶

One batch of 8 images is 8 192 pixel-examples. Applying all of their feedback in a single commit is a mini-batch of 8 192, far past the point where batched feedback stops behaving like the classical one-example-at-a-time algorithm. patches_per_commit caps how many pixels share one commit; None commits a whole batch at once, and 1 is the classical algorithm. The sweep below uses the single-scale model with position literals.

In [ ]:

sweep = {}
for ppc in (32, 128, 512, 2048, None):
    m = train(f"position, ppc={ppc}", x_tr, x_te, position=8, patches_per_commit=ppc)
    sweep[str(ppc)] = RESULTS.pop(f"position, ppc={ppc}")
pd.DataFrame(sweep).T[["pixel_accuracy", "mean_iou", "iou_building", "iou_tree", "seconds"]].round(3)

sweep = {} for ppc in (32, 128, 512, 2048, None): m = train(f"position, ppc={ppc}", x_tr, x_te, position=8, patches_per_commit=ppc) sweep[str(ppc)] = RESULTS.pop(f"position, ppc={ppc}") pd.DataFrame(sweep).T[["pixel_accuracy", "mean_iou", "iou_building", "iou_tree", "seconds"]].round(3)

In our run quality degrades gently from 32 to 512 pixels per commit (mean IoU about 0.945 → 0.935 → 0.914) while training gets 3–4× faster at each step, and then collapses: at 2048 mean IoU is about 0.50, and committing a whole batch at once (None) ends at about 0.14, far below the row prior. Thousands of pixels pushing the same clauses in opposite directions before any of it is applied undo each other's learning. The default of 128 sits on the safe side of that cliff; lower it if learning looks unstable, and check the cost before raising it.

8. Class-balanced feedback¶

A Tsetlin machine has no loss to reweight. The lever for class imbalance is which pixels produce feedback: set_class_feedback_p gives each class a probability of contributing, and tt.data.balanced_class_probabilities equalises them (with a floor, so frequent classes still learn).

In [ ]:

class_p = tt.data.balanced_class_probabilities(y_tr, len(CLASSES), floor=0.05)
print("feedback probability per class:", dict(zip(CLASSES, [round(p, 3) for p in class_p.tolist()])))
balanced = train("+ position + balanced", x_tr, x_te, position=8, class_p=class_p)
pd.DataFrame(RESULTS).T.loc[["+ position literals", "+ position + balanced"], cols].round(3)

class_p = tt.data.balanced_class_probabilities(y_tr, len(CLASSES), floor=0.05) print("feedback probability per class:", dict(zip(CLASSES, [round(p, 3) for p in class_p.tolist()]))) balanced = train("+ position + balanced", x_tr, x_te, position=8, class_p=class_p) pd.DataFrame(RESULTS).T.loc[["+ position literals", "+ position + balanced"], cols].round(3)

The rarest class (tree) now produces feedback from every pixel, road from one pixel in twenty. In our run this raises building IoU a little (about 0.865 → 0.881) but costs tree about 0.12 IoU (0.893 → 0.773), so mean IoU falls. Rebalancing is for a rare class that is being lost; once the model can already separate the classes, it costs accuracy. Check the per-class IoU either way.

Takeaways¶

• Report the floor. On a layout-driven dataset a baseline that never sees a pixel gets 0.88 pixel accuracy. Mean IoU, and the per-class IoU of the hard classes, are the numbers to read.
• Context beats a bigger window. Position literals and an OR-pooled pyramid both add what a 3 × 3 patch cannot see, cost nothing extra here, and compose.
• patches_per_commit is a real hyper-parameter. Dense feedback over thousands of pixels at once destroys learning; keep the default unless you have measured otherwise.
• Every pixel's label has an explanation in named pixels and rows, which a convolutional network does not give you.
• model.eval() before predicting. Empty clauses are True while learning and False when predicting; the train helper above calls it for that reason.

Further reading: the segmentation guide (ignore_index, stride, real datasets such as Pascal VOC, multi-label masks) and notebook 05, which uses these scenes to test the claims of the CTM-UNet paper.