Race a model pool

Train a custom LSTM and a fine-tuned foundation model in the same run — deploy whichever wins.

You rarely know in advance which architecture fits your data. A pipeline's models array is a pool: every run trains all entries on the same segment and horizon, scores them on the same held-out data, and the best score becomes the promotion candidate. This example races a custom LSTM against a fine-tuned foundation model — and lets you make the deployment call yourself.

You'll need: the workspace, segment, and custom model from CSV to live forecast (steps 1–4), and a plan that includes fine-tuning.

1. A pipeline with two contenders

Custom models are referenced by catalog ID; foundation models by family + slug. You never declare what kind a model is — the platform resolves it:

from predictai import PredictAI

client = PredictAI(token="pa_live_...", workspace_id="ws_...")

pipeline = client.pipelines.post_pipelines(json={
    "name": "Sales - LSTM vs predictFM",
    "segment": SEGMENT_ID,
    "workspace_id": client.workspace_id,
    "models": [
        {"model": MODEL_ID},                                # your LSTM
        {"family": "predictfm", "slug": "predictfm-f-v0.1",  # fine-tuned adapter
         "intensity": "adapt"},
    ],
    "forecast_horizon": 14,
    "promotion_policy": {"mode": "manual"},   # you decide who deploys
})
PIPELINE_ID = pipeline["data"]["uid"]

import { PredictAI } from "@predictai/sdk";

const client = new PredictAI({ token: "pa_live_...", workspaceId: "ws_..." });

const pipeline = await client.pipelines.postPipelines({
  json: {
    name: "Sales - LSTM vs predictFM",
    segment: SEGMENT_ID,
    workspace_id: "ws_...",
    models: [
      { model: MODEL_ID },                                 // your LSTM
      { family: "predictfm", slug: "predictfm-f-v0.1",     // fine-tuned adapter
        intensity: "adapt" },
    ],
    forecast_horizon: 14,
    promotion_policy: { mode: "manual" },  // you decide who deploys
  },
});
const PIPELINE_ID: string = pipeline.data.uid;

Every entry is contract-checked at create time — data-sufficiency, label arity, the foundation family's horizon cap — so a model that can't train on this segment is rejected before anything is charged. intensity (light / adapt / deep) trades fine-tuning cost against adaptation.

2. Estimate, then start the race

Foundation trainings charge upfront, so check the price first — then queue a run:

estimate = client.billing.post_billing_estimate(json={
    "operation": "adapter_training.adapt",   # the pool's foundation entry
    "units": 1,
})
print("fine-tune cost:", estimate["data"]["cost_credits"], "credits")

client.pipelines.post_pipelines_by_model_id_train(PIPELINE_ID)

const estimate = await client.billing.postBillingEstimate({
  json: { operation: "adapter_training.adapt", units: 1 },  // the pool's foundation entry
});
console.log("fine-tune cost:", estimate.data.cost_credits, "credits");

await client.pipelines.postPipelinesByModelIdTrain(PIPELINE_ID);

3. Read the scoreboard

One cycle produces one training row per pool entry — same segment, same horizon, same held-out scoring, so the numbers are directly comparable:

import time

while True:
    runs = client.trainings.get_training(params={"model_id": PIPELINE_ID, "per_page": 5})
    rows = runs["data"]["trainings"]
    if all(r["last_status"] in ("completed", "failed") for r in rows[:2]):
        break
    time.sleep(30)

for r in sorted(rows[:2], key=lambda r: r["accuracy_score"] or 0, reverse=True):
    kind = r["training_kind"] or "custom"
    print(f'{kind:12} accuracy={r["accuracy_score"]}  training={r["uid"]}')

winner = max(rows[:2], key=lambda r: r["accuracy_score"] or 0)

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

let rows: any[];
for (;;) {
  const runs = await client.trainings.getTraining({ params: { model_id: PIPELINE_ID, per_page: 5 } });
  rows = runs.data.trainings;
  if (rows.slice(0, 2).every((r) => r.last_status === "completed" || r.last_status === "failed")) break;
  await sleep(30_000);
}

for (const r of rows.slice(0, 2)) {
  console.log(`${r.training_kind || "custom"} accuracy=${r.accuracy_score} training=${r.uid}`);
}

const winner = rows.slice(0, 2)
  .reduce((a, b) => ((a.accuracy_score ?? 0) >= (b.accuracy_score ?? 0) ? a : b));

foundation   accuracy=0.94  training=7d81c3f0-...
custom       accuracy=0.91  training=a4c9d0e7-...

4. Deploy the winner

With promotion_policy: manual, nothing serves until you say so:

deployment = client.deployments.post_deployments(json={
    "pipeline_id": PIPELINE_ID,
    "training_id": winner["uid"],
    "reason": "Won the pool on held-out accuracy",
})
print(deployment["status"])   # "deploying" -> "deployed"

const deployment = await client.deployments.postDeployments({
  json: {
    pipeline_id: PIPELINE_ID,
    training_id: winner.uid,
    reason: "Won the pool on held-out accuracy",
  },
});
console.log(deployment.status);  // "deploying" -> "deployed"

From here, inference is identical to any pipeline — /v1/inference/pipelines/{id}/infer-segment routes to whatever is currently serving.

Why this pattern wins

No architecture bet. Add entries to the pool and every scheduled run re-litigates the choice on fresh data. Yesterday's winner defends its title every cycle.
Apples to apples. Shared segment, shared horizon, shared scoring — the accuracy numbers mean the same thing across kinds.
Cheap to automate. Flip promotion_policy to auto and the winner deploys itself — but only when it beats the currently serving model, so accuracy only ratchets up.

Set a training_schedule and a budget cap, and this becomes a self-improving system you check in on, not one you operate.

1. A pipeline with two contenders

2. Estimate, then start the race

3. Read the scoreboard

4. Deploy the winner

Why this pattern wins

On this page