Predict.aiDocs
Trainings

Concepts

The training lifecycle, how runs are scored, champions, and cost.

Lifecycle

Every run moves through the same states:

queued → started → completed
                 ↘ failed

Three fields track it:

  • last_status — the canonical current state (started, completed, failed; cancelled if an operator stopped the run).
  • status — the full transition history, an array of { "action", "timestamp" } entries starting with queued.
  • progress — while a run is in flight, a live { "stage", "pct", "epoch", "total_epochs" } snapshot.

A run appears in the list the instant it's queued — if the worker pool is cold, it can sit in queued for a few seconds before started_at is stamped. For live updates without polling, subscribe to the workspace's realtime channel for training.job.* events.

The report

Every completed run carries a scored report. Two numbers matter for every training kind:

FieldQuestion it answers
accuracy_scoreHow close were predictions to actuals? (WMAPE-derived headline)
skill_scoreDoes the model beat a constant baseline? (R²-based; catches models that just predict the mean)

The report also includes individual_comparisons — ground-truth vs predicted series (capped at 365 points each) so you can eyeball the fit — and kind-specific detail:

  • Custom models track per-epoch metrics: report.epochs, final_loss, and an epochs_status table. has_epochs tells you whether they exist.
  • Foundation trainings have no epoch loop. Their score is avg_train_loss (lower is better), alongside rank, size_mb, and held-out validation metrics (val_mse, val_mae).

Champions

The champion is the best-scoring completed training for a pipeline or goal — the run that deserves to serve live traffic. Every run is compared against the current best; the pipeline's promotion policy (or a goal's tournament) decides what happens when a new run wins. The full mental model lives in Trainings & champions.

ready_for_promotion: true on a run means it's eligible to deploy; the deployments array shows where it's currently serving (empty means not deployed). Deploy any completed run — champion or not — via Deployments.

Cost

Every run records what it consumed: cost is denominated in credits (cost_unit: "credits", never USD). Foundation trainings charge upfront based on training intensity; custom-code trainings bill by elapsed compute after the run. Rates and estimates live in Billing.

A run that fails before doing real work — for example, insufficient credits at dispatch — still leaves a record: last_status: "failed" with the reason in message.

On this page