Skip to content

Train models with JAX and Flax.

Language models, diffusion and JEPA, on one device or a mesh.

curl -LsSf https://dewml.dev/install.sh | sh

Trained with Dew

176M hybrid DiT · Prompts and settings

green and purple northern lights reflected in a frozen lake, snowy mountains at night, vivid colors, dramatic lighting, highly detailed
green and purple northern lights reflected in a frozen lake, snowy mountains at night, vivid colors, dramatic lighting, highly detailed
a mountain lake reflecting snowy peaks, vivid colors, dramatic lighting, highly detailed
a mountain lake reflecting snowy peaks, vivid colors, dramatic lighting, highly detailed
a lighthouse on a rocky coast at sunset, dramatic waves and golden light, vivid colors, dramatic lighting, highly detailed
a lighthouse on a rocky coast at sunset, dramatic waves and golden light, vivid colors, dramatic lighting, highly detailed
a snowy mountain village at dusk, warm glowing windows and blue snow, vivid colors, dramatic lighting, highly detailed
a snowy mountain village at dusk, warm glowing windows and blue snow, vivid colors, dramatic lighting, highly detailed
a canyon with towering red rock cliffs and a winding river at sunset
a canyon with towering red rock cliffs and a winding river at sunset
a Japanese temple beside a cherry blossom tree on a spring morning, vivid colors, dramatic lighting, highly detailed
a Japanese temple beside a cherry blossom tree on a spring morning, vivid colors, dramatic lighting, highly detailed
a cozy wooden cabin in a snowy forest at night, warm light in the windows, vivid colors, dramatic lighting, highly detailed
a cozy wooden cabin in a snowy forest at night, warm light in the windows, vivid colors, dramatic lighting, highly detailed
a canal in Venice at sunset with gondolas and old buildings
a canal in Venice at sunset with gondolas and old buildings
colorful hot air balloons floating above a green valley at sunrise
colorful hot air balloons floating above a green valley at sunrise
a red maple tree beside a calm lake in autumn, mountains in the background, vivid colors, dramatic lighting, highly detailed
a red maple tree beside a calm lake in autumn, mountains in the background, vivid colors, dramatic lighting, highly detailed
colorful fireworks over a harbor city at night, reflections in the water
colorful fireworks over a harbor city at night, reflections in the water
rolling sand dunes in the desert at sunset, deep orange sand and purple sky, vivid colors, dramatic lighting, highly detailed
rolling sand dunes in the desert at sunset, deep orange sand and purple sky, vivid colors, dramatic lighting, highly detailed
a turquoise alpine lake surrounded by pine trees and rugged mountains
a turquoise alpine lake surrounded by pine trees and rugged mountains
a dramatic sunset over the ocean, fiery orange and pink clouds
a dramatic sunset over the ocean, fiery orange and pink clouds
a lush green valley with a winding river and white clouds, Studio Ghibli style landscape, vivid colors, dramatic lighting, highly detailed
a lush green valley with a winding river and white clouds, Studio Ghibli style landscape, vivid colors, dramatic lighting, highly detailed
a castle on a rocky hill above a misty valley at sunrise, vivid colors, dramatic lighting, highly detailed
a castle on a rocky hill above a misty valley at sunrise, vivid colors, dramatic lighting, highly detailed
a waterfall in a lush green forest, vivid colors, dramatic lighting, highly detailed
a waterfall in a lush green forest, vivid colors, dramatic lighting, highly detailed
a waterfall in a lush green forest, vivid colors, dramatic lighting, highly detailed
a waterfall in a lush green forest, vivid colors, dramatic lighting, highly detailed

Latest tutorials and posts

  1. PostA racing drone that learns to fly from its cameraA new tutorial, A racing drone that flies from its camera, trains a first-person racing drone from scratch.
  2. TutorialContinual learning in contextA trained language model is frozen: whatever it reads, its weights stay where training left them.
  3. PostHow Dew works, from a model to kernels on the GPUA new page, How Dew works, follows one training run through Dew, from the Flax model and data you write down to the kernels that run on the GPU, and checks each step against the code.
  4. TutorialWorld models: acting and imagining in one loopSuppose you have taught a program to balance a pole on a cart by letting it practise in a simulator, pushing the cart left and right millions of times until it stops dropping the pole.

Video

Wan 2.1 1.3B's released weights, sampled with Dew's JAX implementation · Prompts and settings

A slow aerial shot gliding over a misty pine forest at sunrise, golden light breaking through the fog between the trees, cinematic, highly detailed
Ocean waves crashing against dark volcanic rocks at sunset, the spray glowing orange in the backlight, slow motion, cinematic
A hummingbird hovering beside bright red flowers in a sunlit garden, its wings a soft blur, shallow depth of field, nature documentary footage
Green northern lights rippling across a starry night sky above a snowy mountain lake, the aurora reflected and shimmering on the still water

Training

A small decoder trains on Tiny Shakespeare on one A100, then writes one sample from each of the prompts ROMEO: and JULIET:.

train.py, editable

Run runs the cell on a shared 4-vCPU host; training stops after 20 steps.

python train.py
╭─  dew · CausalTransformer  ──────────────────────────────────────────────────────────────────╮
│ 10.7M parameters  ·  on 1 × NVIDIA A100-SXM4-40GB  ·  batch 64  ·  float32                   │
│                                                                                              │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1000/1000  100%  33.5 step/s  0:00:46 │
│ data 64,192 rows  ·  0 rejected                                                              │
│                                                                                              │
│ train                                                                                        │
│   loss                    1.201  █▅▄▃▃▂▂▁▂▁      ↓ 40%                                       │
│   ce                      1.201  █▅▄▃▃▂▂▁▂▁      ↓ 40%                                       │
│   perplexity              3.322  █▅▄▂▃▁▂▁▂▁      ↓ 56%                                       │
│   token_accuracy          62.4%  ▁▄▅▇▆█▇███  ↑ 21.9 pt                                       │
│   data_wait_ms          0.01949  ▄▅▅█▇▂▅▁▂▅     ↑ 3.4%                                       │
│   grad_norm              0.2868  █▄▂▂▂▁▁▁▁▁      ↓ 72%                                       │
│   cache_misses            3.000  ▁█████████      ↑ 50%                                       │
│   compiles 1.000   cache_hits 9.000   peak_memory_gib 4.016                                  │
│ throughput                                                                                   │
│   step_time_ms            29.77  █▅▁▂▂▅▂▂▂▂     ↓ 2.6%                                       │
│   samples_per_sec         2,150  ▁▄█▇▇▄▇▇▇▇     ↑ 2.6%                                       │
│   tokens                 16,076  ▄▁▅▃█▄▇▄█▄          →                                       │
│   tokens_per_s          539,918  ▁▃█▇█▄█▇█▇     ↑ 2.6%                                       │
│   mfu                     11.7%  ▁▃█▇█▄█▇█▇   ↑ 0.3 pt                                       │
│   hlo_flops_util ≥        12.6%  ▁▄█▇▇▄▇▇▇▇   ↑ 0.3 pt                                       │
│   hfu ≥                   12.6%  ▁▄█▇▇▄▇▇▇▇   ↑ 0.3 pt                                       │
│   uncounted kernels 50.00                                                                    │
│ optimizer                                                                                    │
│   learning_rate       1.000e-04  ██▇▇▆▅▄▃▂▁      ↓ 90%                                       │
│   accepted 100.0%                                                                            │
│                                                                                              │
│ val · step 1000                                                                              │
│   perplexity       3.567  █▁     ↓ 6.6%                                                      │
│   loss             1.272  █▁     ↓ 5.1%                                                      │
╰───────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯
╭─  ✓ CausalTransformer  ────────────────────────────────────────╮
│     trained  1000 steps in 0:00:50                             │
│  first step  after 19.90 s                                     │
│        then  33.5 step/s                                       │
│     goodput  59.9% of the wall time in steps                   │
│  final loss  1.201                                             │
│ val at 1000  perplexity 3.567 ↓ 6.6% █▁   loss 1.272 ↓ 5.1% █▁ │
╰────────────────────────────────────────────────────────────────╯
ROMEO:
And therefore, he shall be the whole of the sea,
Of her that makes the warm of such a fair court,
As as the proclaims to my brother's love,
Or make my executioner than the common
That he be so much a

JULIET:
Then she is God's name, come, that say the people
Is the requirest of it.

Recorded on one A100 with JAX 0.11.2.post11 and Dew d2adda0 on 2026-10-21. The run took 65 s. The replay lasts about 10 s. The terminal was 96×200; scroll the panel for earlier lines. The replay needs JavaScript; with reduced motion you see the last frame. Original recording.

Full script

Language modeling

The objective creates the model's variables, computes the loss and runs evaluation. Trainer.fit steps the optimizer, logs progress and saves checkpoints, and every model on this page trains through that one call.

Language modeling, editable

Run runs the cell on a shared 4-vCPU host; training stops after 20 steps.

python lm.py
╭─  dew · CausalTransformer  ──────────────────────────────────────────────────────────────────╮
│ 18,560 parameters  ·  on 1 × cpu  ·  batch 8  ·  float32                                     │
│                                                                                              │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  3/3  100%  0:00:03 │
│ data 48 rows  ·  0 rejected                                                                  │
│                                                                                              │
│ train                                                                                        │
│   loss                  5.874  ▁                                                             │
│   ce                    5.874  ▁                                                             │
│   perplexity            355.7  ▁                                                             │
│   token_accuracy         0.4%  ▁                                                             │
│   data_wait_ms        0.02289  ▁                                                             │
│   grad_norm             1.819  ▁                                                             │
│   compiles              1.000  ▁                                                             │
│   cache_hits                0  ▁                                                             │
│   cache_misses          5.000  ▁                                                             │
│ throughput                                                                                   │
│   step_time_ms          83.48  ▁                                                             │
│   samples_per_sec       95.83  ▁                                                             │
│   tokens                512.0  ▁                                                             │
│   tokens_per_s          6,133  ▁                                                             │
│ optimizer                                                                                    │
│   accepted             100.0%  ▁                                                             │
│   learning_rate      0.001000  ▁                                                             │
╰───────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯
╭─  ✓ CausalTransformer  ────────────────────╮
│    trained  3 steps in 0:00:04             │
│ first step  after 4.23 s                   │
│       then  29.3 step/s                    │
│    goodput  1.6% of the wall time in steps │
│ final loss  5.874                          │
╰────────────────────────────────────────────╯

Recorded on a 4-vCPU Cloudflare container with JAX 0.11.2.post11 and Dew d2adda0 on 2026-10-21. The run took 35 s.

Full script

Representation learning

A JEPA encoder sees part of an image and learns to predict the representations of the patches it cannot see.

JEPA, editable

Run runs the cell on a shared 4-vCPU host; training stops after 20 steps.

python jepa.py
╭─  dew · JepaEncoder  ────────────────────────────────────────────────────────────────────────╮
│ 18,736 parameters  ·  on 1 × cpu  ·  batch 64  ·  float32                                    │
│                                                                                              │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  3/3  100%  0:00:09 │
│ data 384 rows  ·  0 rejected                                                                 │
│                                                                                              │
│ train                                                                                        │
│   loss                   1.489  ▁                                                            │
│   repr_cov_offdiag      0.1102  ▁                                                            │
│   repr_std              0.3723  ▁                                                            │
│   data_wait_ms         0.02631  ▁                                                            │
│   grad_norm              2.365  ▁                                                            │
│   compiles               1.000  ▁                                                            │
│   cache_hits                 0  ▁                                                            │
│   cache_misses           5.000  ▁                                                            │
│ throughput                                                                                   │
│   step_time_ms           207.3  ▁                                                            │
│   samples_per_sec        308.8  ▁                                                            │
│ optimizer                                                                                    │
│   accepted              100.0%  ▁                                                            │
│   learning_rate       0.001000  ▁                                                            │
╰───────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯
╭─  ✓ JepaEncoder  ──────────────────────────╮
│    trained  3 steps in 0:00:13             │
│ first step  after 12.66 s                  │
│       then  17.9 step/s                    │
│    goodput  0.9% of the wall time in steps │
│ final loss  1.489                          │
╰────────────────────────────────────────────╯

Recorded on a 4-vCPU Cloudflare container with JAX 0.11.2.post11 and Dew d2adda0 on 2026-10-21. The run took 46 s.

Fine-tuning

Load a Hugging Face checkpoint as a Flax model, fine-tune it with the same trainer, and save it back in the Hugging Face layout with its original tensor names. The first cell fine-tunes SmolLM2-135M on Tiny Shakespeare and compares its text before and after.

Fine-tune SmolLM2-135M, editable

Run runs the cell on a shared 4-vCPU host; training stops after 20 steps.

python finetune.py
╭─  dew · CausalTransformer  ──────────────────────────────────────────────────────────────────╮
│ 134.5M parameters  ·  on 1 × cpu  ·  batch 1  ·  float32                                     │
│                                                                                              │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  20/20  100%  1.4 step/s  0:00:35 │
│ data 23 rows  ·  0 rejected                                                                  │
│                                                                                              │
│ train                                                                                        │
│   loss                  3.218  █▅▆▁      ↓ 27%                                               │
│   ce                    3.218  █▅▆▁      ↓ 27%                                               │
│   perplexity            24.97  █▄▆▁      ↓ 69%                                               │
│   token_accuracy        40.6%  ▁▄▇█  ↑ 14.1 pt                                               │
│   data_wait_ms          1.059  ▁▁██      ↑ 35×                                               │
│   grad_norm             9.280  █▁▂▁      ↓ 17%                                               │
│   cache_misses          12.00  ▁███     ↑ 9.1%                                               │
│   compiles 1.000   cache_hits 0                                                              │
│ throughput                                                                                   │
│   step_time_ms          767.5  █▂▁▂      ↓ 29%                                               │
│   samples_per_sec       1.303  ▁▇█▇      ↑ 40%                                               │
│   tokens                64.00  ██▁█          →                                               │
│   tokens_per_s          83.39  ▁█▇█      ↑ 40%                                               │
│ optimizer                                                                                    │
│   accepted 100.0%   learning_rate 1.000e-04                                                  │
╰───────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯
╭─  ✓ CausalTransformer  ─────────────────────╮
│    trained  20 steps in 0:00:37             │
│ first step  after 22.22 s                   │
│       then  1.3 step/s                      │
│    goodput  39.5% of the wall time in steps │
│ final loss  3.218                           │
╰─────────────────────────────────────────────╯
Before:

I'm glad you're interested in learning more about the history of the Eiffel Tower. The tower was
 built in 1889 as a
After:

BUCKINGHAM:

Hastings:

BUCINGHAM:

BUCINGHAM:

Recorded on a 4-vCPU Cloudflare container with JAX 0.11.2.post11 and Dew d2adda0 on 2026-10-21. The run took 122 s.

A larger model: Qwen3-0.6B, one step, then saved in its original layout.

Fine-tune Qwen3-0.6B, editableRun in Colab

This cell needs more memory than a shared host has, so Run in Colab opens it on a GPU.

python pretrained.py
 Paris. The capital of France is also the capital of the
╭─  dew · CausalTransformer  ──────────────────────────────────────────────────────────────────╮
│ 596.0M parameters  ·  on 1 × cpu  ·  batch 1  ·  bfloat16                                    │
│                                                                                              │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/1  100%  0:00:39 │
│ data 4 rows  ·  0 rejected                                                                   │
│                                                                                              │
│ train                                                                                        │
│   loss                  3.576  ▁                                                             │
│   ce                    3.576  ▁                                                             │
│   perplexity            35.73  ▁                                                             │
│   token_accuracy        45.3%  ▁                                                             │
│   data_wait_ms              0  ▁                                                             │
│   grad_norm             65.00  ▁                                                             │
│   compiles              1.000  ▁                                                             │
│   cache_hits                0  ▁                                                             │
│   cache_misses          13.00  ▁                                                             │
│ throughput                                                                                   │
│   step_time_ms          9,160  ▁                                                             │
│   samples_per_sec      0.1092  ▁                                                             │
│   tokens                64.00  ▁                                                             │
│   tokens_per_s          6.987  ▁                                                             │
│ optimizer                                                                                    │
│   accepted             100.0%  ▁                                                             │
│   learning_rate     1.000e-05  ▁                                                             │
╰───────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯
╭─  ✓ CausalTransformer  ────────────────────╮
│    trained  1 steps in 0:00:41             │
│ first step  after 40.70 s                  │
│    goodput  0.1% of the wall time in steps │
│ final loss  3.576                          │
╰────────────────────────────────────────────╯

Recorded on a 4-vCPU Cloudflare container with JAX 0.11.2.post11 and Dew d2adda0 on 2026-10-21. The run took 138 s.

Supported families: Qwen, Llama, Gemma, Mistral, DeepSeek, Mamba and multimodal decoders.

Inference

A TextGeneration task holds the model, its weights and the sampling settings. Run the cell to generate on a shared CPU host.

generate.py
 Paris. Paris is the largest city in France and the capital of the French department of the Espace de Paris.

Recorded on a 4-vCPU Cloudflare container with Dew aee614a on 2026-10-08, in 40 s. Run the cell on a live CPU kernel.

Server runs many requests through one task in shared batches, and each request keeps its own random key and KV cache:

Batched decoding, editableRun in Colab

This cell needs more memory than a shared host has, so Run in Colab opens it on a GPU.

python serving.py
[' Paris. The capital of France is also the capital of the Republic of France. The capital of Fr
ance is also the capital', ' Tokyo. The capital of the United States is Washington, D.C. The cap
ital of France is Paris. The capital of']

Recorded on a 4-vCPU Cloudflare container with JAX 0.11.2.post3 and Dew aee614a on 2026-10-08. The run took 64 s.

Store weights in int8 or fp8:

Weight-only int8 and fp8, editableRun in Colab

This cell needs more memory than a shared host has, so Run in Colab opens it on a GPU.

python formats.py
int8  Paris. The capital of Italy is Rome. The capital of
fp8  Paris. The capital of France is also the capital of the

Recorded on a 4-vCPU Cloudflare container with JAX 0.11.2.post4 and Dew a758210 on 2026-10-19. The run took 79 s.

The serving guide covers the paged KV cache, prefix sharing, constrained decoding, and the error each quantized format adds.

Decision models

A decision model answers typed questions about a state in one forward pass: each question names its options, and the answer is a probability distribution over exactly those options. Laya, a ModernBERT-large decision model, loads as a Dew model, and systemone takes and returns Jev's wire format.

Route a support ticket, editable

Run runs the cell on a shared 4-vCPU host; training stops after 20 steps.

python decide.py
{
  "department": {
    "type": "choice",
    "choice": "billing",
    "probabilities": {
      "billing": 0.9897,
      "technical": 0.0103
    },
    "confidence": 0.9794
  },
  "urgency": {
    "type": "score",
    "score": 1.7091,
    "legend": {
      "0": "not urgent",
      "1": "soon",
      "2": "blocking"
    },
    "probabilities": {
      "0": 0.0483,
      "1": 0.1942,
      "2": 0.7574
    },
    "confidence": 0.5636
  },
  "churn_risk": {
    "type": "noul",
    "noul": 0.0818
  }
}

Recorded on a 4-vCPU Cloudflare container with JAX 0.11.2.post3 and Dew aee614a on 2026-10-08. The run took 62 s.

The decision models guide covers calibration, abstention and fine-tuning a decision head on any backbone.

Diffusion

The process sets how training adds noise and what the model learns to predict. At generation time a solver runs the process in reverse, and you can switch solvers without retraining.

sample.py, editable

Run sends the cell's Dew calls to a process on the host that already holds the model. The same cell runs as is with Dew installed.

The image this cell drew when it was recorded.

Run opens an isolated Python context backed by the shared Dew model on a 4-vCPU CPU host.

Recorded on one A100 with Dew 7575f1a on 2026-10-22, in 93 s.

Or open it in Colab to run it on a GPU and change anything.

Define and train a DiT, editable

Run runs the cell on a shared 4-vCPU host; training stops after 20 steps.

python diffusion.py
╭─  dew · SimpleDiT  ──────────────────────────────────────────────────────────────────────────╮
│ 26,320 parameters  ·  on 1 × cpu  ·  batch 64  ·  float32                                    │
│                                                                                              │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  3/3  100%  0:00:06 │
│ data 384 rows  ·  0 rejected                                                                 │
│                                                                                              │
│ train                                                                                        │
│   loss                 0.6171  ▁                                                             │
│   data_wait_ms        0.03363  ▁                                                             │
│   grad_norm            0.1143  ▁                                                             │
│   compiles              1.000  ▁                                                             │
│   cache_hits                0  ▁                                                             │
│   cache_misses          5.000  ▁                                                             │
│ throughput                                                                                   │
│   step_time_ms          147.9  ▁                                                             │
│   samples_per_sec       432.8  ▁                                                             │
│ optimizer                                                                                    │
│   accepted             100.0%  ▁                                                             │
│   learning_rate      0.001000  ▁                                                             │
╰───────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯
╭─  ✓ SimpleDiT  ────────────────────────────╮
│    trained  3 steps in 0:00:08             │
│ first step  after 7.89 s                   │
│       then  9.9 step/s                     │
│    goodput  2.5% of the wall time in steps │
│ final loss  0.6171                         │
╰────────────────────────────────────────────╯

Recorded on a 4-vCPU Cloudflare container with JAX 0.11.2.post11 and Dew d2adda0 on 2026-10-21. The run took 37 s.

A 176M-parameter text-to-image model at 256×171, trained with Dew. Weights: dewml/hybrid-dit-176m.

the northern lights over a frozen lake at night
“the northern lights over a frozen lake at night”
a lighthouse on a rocky coast at sunset, dramatic waves and golden light
“a lighthouse on a rocky coast at sunset, dramatic waves and golden light”
a snowy mountain village at dusk, warm glowing windows and blue snow, vivid colors, dramatic lighting, highly detailed
“a snowy mountain village at dusk, warm glowing windows and blue snow, vivid colors, dramatic lighting, highly detailed”

A style LoRA

Stable Diffusion 1.5 with a rank-16 LoRA on its attention projections, trained with Dew for 160 steps on 20 public-domain Hiroshige prints from the Library of Congress, on one A100. Each pair shares its prompt and seed; drag the slider to compare. Prompts, seeds and settings.

Before LoRA: a picture in dewprint style of a mountain lake with a small sailboat, snowy peaks, dawnAfter LoRA: a picture in dewprint style of a mountain lake with a small sailboat, snowy peaks, dawn
BeforeAfter
a picture in dewprint style of a mountain lake with a small sailboat, snowy peaks, dawn
Seed 1042
Before LoRA: a picture in dewprint style of a winding river through a village in autumn, red maple treesAfter LoRA: a picture in dewprint style of a winding river through a village in autumn, red maple trees
BeforeAfter
a picture in dewprint style of a winding river through a village in autumn, red maple trees
Seed 2073
Before LoRA: a picture in dewprint style of a wooden bridge in the rain, people crossing with umbrellasAfter LoRA: a picture in dewprint style of a wooden bridge in the rain, people crossing with umbrellas
BeforeAfter
a picture in dewprint style of a wooden bridge in the rain, people crossing with umbrellas
Seed 3104
Before LoRA: a picture in dewprint style of a coastal fishing village, a red sunset over the seaAfter LoRA: a picture in dewprint style of a coastal fishing village, a red sunset over the sea
BeforeAfter
a picture in dewprint style of a coastal fishing village, a red sunset over the sea
Seed 4135
Before LoRA: a picture in dewprint style of a snow-covered pine tree beside a quiet templeAfter LoRA: a picture in dewprint style of a snow-covered pine tree beside a quiet temple
BeforeAfter
a picture in dewprint style of a snow-covered pine tree beside a quiet temple
Seed 5166
Before LoRA: a picture in dewprint style of cherry blossom trees beside a river, Mount Fuji in the distanceAfter LoRA: a picture in dewprint style of cherry blossom trees beside a river, Mount Fuji in the distance
BeforeAfter
a picture in dewprint style of cherry blossom trees beside a river, Mount Fuji in the distance
Seed 6197

Distributed

To use more devices, give Trainer a mesh; the objective and data stay as they are. FSDP shards parameters and optimizer state, tensor parallelism splits each layer's weight matrices, and an expert axis puts an MoE's experts on different devices. The devices left over form the data-parallel axis.

Four simulated CPU devices, editable

Run runs the cell on a shared 4-vCPU host; training stops after 20 steps.

python mesh.py
╭─  dew · CausalTransformer  ──────────────────────────────────────────────────────────────────╮
│ 18,560 parameters  ·  on 4 × cpu  ·  mesh fsdp 2 × tensor 2, 0% of the parameters' bytes spl │
│                                                                                              │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  3/3  100%  0:00:03 │
│ data 48 rows  ·  0 rejected                                                                  │
│                                                                                              │
│ train                                                                                        │
│   loss                  5.874  ▁                                                             │
│   ce                    5.874  ▁                                                             │
│   perplexity            355.7  ▁                                                             │
│   token_accuracy         0.4%  ▁                                                             │
│   data_wait_ms        0.03056  ▁                                                             │
│   grad_norm             1.819  ▁                                                             │
│   compiles              1.000  ▁                                                             │
│   cache_hits                0  ▁                                                             │
│   cache_misses          5.000  ▁                                                             │
│ throughput                                                                                   │
│   step_time_ms          117.4  ▁                                                             │
│   samples_per_sec       68.15  ▁                                                             │
│   tokens                512.0  ▁                                                             │
│   tokens_per_s          4,362  ▁                                                             │
│ optimizer                                                                                    │
│   accepted             100.0%  ▁                                                             │
│   learning_rate      0.001000  ▁                                                             │
╰───────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯
╭─  ✓ CausalTransformer  ────────────────────╮
│    trained  3 steps in 0:00:04             │
│ first step  after 4.16 s                   │
│       then  15.0 step/s                    │
│    goodput  3.1% of the wall time in steps │
│ final loss  5.874                          │
╰────────────────────────────────────────────╯

Recorded on a 4-vCPU Cloudflare container with JAX 0.11.2.post11 and Dew d2adda0 on 2026-10-21. The run took 35 s.

Placement and sharding · Several hosts · Cloud TPU

Expert routing

With the experts split over devices, each MoE layer's router sends tokens to experts on other devices through all-to-all exchanges. This is the routing examples/moe_mesh.py recorded over 300 training steps.

training step 1 of 300

Eight simulated CPU devices: data 2 × expert 2 × FSDP 2. Arrows show the recorded assignments sent between devices; each circle shows the assignments kept locally. Full routing record.

Post-training

GRPO samples a group of completions for each prompt and scores them with a reward function. The reward here is the fraction of adjacent tokens that count up by one, modulo 13.

Rollout and reward, editable

Run runs the cell on a shared 4-vCPU host; training stops after 20 steps.

python grpo.py
╭─  dew · CausalTransformer  ──────────────────────────────────────────────────────────────────╮
│ 83,136 parameters  ·  on 1 × cpu  ·  batch 8  ·  float32                                     │
│                                                                                              │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  150/150  100%  27.5 step/s  0:00:12 │
│ data 1,224 rows  ·  0 rejected                                                               │
│                                                                                              │
│ train                                                                                        │
│   loss                       0.04121  █▅▆▃▄▇▅▃▅▅▅█▄▂▄▄█▃▂▃▂▁▃▄▃▃▅▅▆▅▅▅▂▅▅▁▅▆▆▃█▂▆     ↓ 7.0% │
│   actor/ppo_kl             4.220e-08  ▅▃█▃▄▄▆▅▂▆▅▇▄▆▃█▃▄▅▇▆▅▆▆▄▆▄▇▅▂▅▄▄▆▇▆▁▄▇▆▅▅▇      ↑ 12× │
│   kl                           2.061  █▅▆▃▄▇▅▃▅▅▅█▄▂▄▄█▃▂▃▂▁▃▄▃▃▅▅▆▅▅▅▂▅▅▁▅▆▆▃█▂▆     ↓ 7.0% │
│   pg                       3.353e-08  █▅▄▄▄▇▁▅▄▄▆█▆▄▄▃▅▄▅▄▆▄▆▄▄▁▅▄▄▂▅▄▄▄▄▄▃▄▄▄▃▄▆      ↓ 62% │
│   data_wait_ms               0.02828  ▃▂▃█▃▄██▃█▇▃▅▁▄▄▃▂▂▁▃▇▃▁▃▆█▃▄▁▁▁▃▂▂▃▁▄▂▃▄▃▃     ↓ 4.5% │
│   grad_norm                   0.3026  ▇▆▂▂▂▆▆▆▆▂▇█▇▂▁▇▇▁▆▁▆▃█▁▂▆▇▁▁▆▆▁▁▁▁▁▇▂▁▂▇▃▇      ↓ 45% │
│   actor/pg_clipfrac 0   actor/pg_clipfrac_lower 0   mismatch/ess 1.000   mismatch/k3_kl 0    │
│ mismatch/kl 0   compiles 1.000   cache_hits 0   cache_misses 21.00                           │
│ throughput                                                                                   │
│   step_time_ms                 11.05  ▅▅▄▅▃█▆█▅▅▅▇▅▅▄▇▅▄▅▄▇▄▅▄▄▇▅▅▅▆▅▄▅▅▆▅▆▅▇▅▅▆▁      ↓ 63% │
│   samples_per_sec              2,896  ▄▄▅▄▆▁▃▁▄▄▄▂▄▄▅▂▄▅▄▅▂▅▄▅▅▂▄▄▄▃▄▅▄▄▃▄▃▄▂▄▄▃█     ↑ 169% │
│   rollout_seconds            0.03291  ▆▂▂▁▂▂▂▅█▃▃▃▆▂▃▁▆▂▁▂▁▆▁▂▁▁▅▁▂▂▅▂▁▂▂▄▃▄▃▆▂▃▃      ↓ 34% │
│ optimizer                                                                                    │
│   accepted 100.0%   learning_rate 0.001000                                                   │
│ rollout                                                                                      │
│   latency/max                0.01380  █▂▂▁▂▄▂▄▇▂▃▃▇▃▃▂▃▃▂▃▂█▂▃▂▂▃▂▂▂▇▂▁▃▂▁▃▄▃▇▂▃▃      ↓ 58% │
│   latency/p50                0.01340  █▂▂▁▂▃▂▄▇▂▃▃▇▃▃▂▃▃▂▃▂█▂▂▂▂▃▂▂▂▇▂▁▃▂▂▃▄▃▇▂▃▃      ↓ 59% │
│   latency/p90                0.01375  █▂▂▁▂▃▂▄▇▂▃▃▇▃▃▂▃▃▂▃▂█▂▃▂▂▃▂▂▂▇▂▁▃▂▁▃▄▃▇▂▃▃      ↓ 58% │
│   latency/p99                0.01379  █▂▂▁▂▄▂▄▇▂▃▃▇▃▃▂▃▃▂▃▂█▂▃▂▂▃▂▂▂▇▂▁▃▂▁▃▄▃▇▂▃▃      ↓ 58% │
│   reward/mean                 0.8527  ▄▆▃▃▇▇▃▆▅▅▅▄▅▅▅▃▃▅▆▇▁▅▃▇▃▃▅▃██▄▅▇▅▅▅▄█▃▂▇▅█     ↑ 5.5% │
│   reward/observed             0.8527  ▄▆▃▃▇▇▃▆▅▅▅▄▅▅▅▃▃▅▆▇▁▅▃▇▃▃▅▃██▄▅▇▅▅▅▄█▃▂▇▅█     ↑ 5.5% │
│   waited_seconds             0.01533  █▃▂▁▂▄▂▄▇▃▃▃▇▃▃▂▃▃▂▃▂█▂▃▂▂▄▂▂▂▇▂▁▃▂▁▃▄▃▇▂▃▄      ↓ 55% │
│   groups 8.000   lag 0   lag/max 0   lag/mean 0   length/mean 8.000   masked/cancelled 0     │
│ masked/infra_error 0   merge/calls_per_chain 1.000   pack/fill 100.0%   resubmitted 0        │
│ status/agent_error 0.0%   status/cancelled 0.0%   status/completed 0.0%   status/infra_error │
│ 0.0%   status/truncated 100.0%                                                               │
╰───────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯
╭─  ✓ CausalTransformer  ─────────────────────╮
│    trained  150 steps in 0:00:14            │
│ first step  after 8.67 s                    │
│       then  27.8 step/s                     │
│    goodput  38.1% of the wall time in steps │
│ final loss  0.04121                         │
╰─────────────────────────────────────────────╯

Recorded on a 4-vCPU Cloudflare container with JAX 0.11.2.post16 and Dew 8b4627a on 2026-10-24. The run took 33 s.

10step 1step 150
Mean reward of the sampled completions at each of the recorded run's 150 updates (raw log). SFT, DPO, PPO and Flow-GRPO train with the same Trainer.

Reinforcement learning from pixels

A racing drone learns to fly through gates from a 64-pixel camera image and what its sensors read, with no demonstrations and no privileged state: a Dreamer world model learns from its own flights, and the policy learns inside the model's imagination. The film shows four drones of the trained policy, each through its own camera.

Averaged over the drones of one race from the same starts, the trained policy passes 44.4 gates in a 2000-step episode without a crash; the untrained one passes 0 and crashes after 515 steps. From held-out training starts the trained policy passes 36.2.

A racing drone that flies from its camera · Recording and scores

Reliability

A checkpoint stores the training state and the position in the data stream. When the job receives SIGTERM, fit stops at the next step every process agrees on, writes a checkpoint and exits with status 143. Run the same command again to resume.

Save and resume, editable

Run runs the cell on a shared 4-vCPU host; training stops after 20 steps.

python reliability.py
╭─  dew · CausalTransformer  ──────────────────────────────────────────────────────────────────╮
│ 18,560 parameters  ·  on 1 × cpu  ·  batch 8  ·  float32                                     │
│                                                                                              │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  2/2  100%  3.5 step/s  0:00:03 │
│ data 40 rows  ·  0 rejected                                                                  │
│                                                                                              │
│ train                                                                                        │
│   loss                  6.108  ▁                                                             │
│   ce                    6.108  ▁                                                             │
│   perplexity            449.6  ▁                                                             │
│   token_accuracy         0.4%  ▁                                                             │
│   data_wait_ms        0.02359  ▁                                                             │
│   grad_norm             1.805  ▁                                                             │
│   compiles              1.000  ▁                                                             │
│   cache_hits                0  ▁                                                             │
│   cache_misses          5.000  ▁                                                             │
│ throughput                                                                                   │
│   step_time_ms          236.4  ▁                                                             │
│   samples_per_sec       33.85  ▁                                                             │
│   tokens                512.0  ▁                                                             │
│   tokens_per_s          2,166  ▁                                                             │
│ optimizer                                                                                    │
│   accepted             100.0%  ▁                                                             │
│   learning_rate      0.001000  ▁                                                             │
╰───────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯
╭─  ✓ CausalTransformer  ────────────────────╮
│    trained  2 steps in 0:00:04             │
│ first step  after 3.39 s                   │
│       then  5.9 step/s                     │
│    goodput  4.3% of the wall time in steps │
│ final loss  6.108                          │
╰────────────────────────────────────────────╯
Resumed from step 2 in /tmp/cell-reliability-0p43xk5u/checkpoints
╭─  dew · CausalTransformer  ──────────────────────────────────────────────────────────────────╮
│ 18,560 parameters  ·  on 1 × cpu  ·  batch 8  ·  float32                                     │
│                                                                                              │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  3/3  100%  0:00:01 │
│ data 48 rows  ·  0 rejected                                                                  │
│                                                                                              │
│ train                                                                                        │
│   loss                  5.874  ▁                                                             │
│   ce                    5.874  ▁                                                             │
│   perplexity            355.7  ▁                                                             │
│   token_accuracy         0.4%  ▁                                                             │
│   data_wait_ms              0  ▁                                                             │
│   grad_norm             1.846  ▁                                                             │
│   compiles              1.000  ▁                                                             │
│   cache_hits            1.000  ▁                                                             │
│   cache_misses          5.000  ▁                                                             │
│ throughput                                                                                   │
│   step_time_ms          62.06  ▁                                                             │
│   samples_per_sec       128.9  ▁                                                             │
│   tokens                512.0  ▁                                                             │
│   tokens_per_s          8,250  ▁                                                             │
│ optimizer                                                                                    │
│   accepted             100.0%  ▁                                                             │
│   learning_rate      0.001000  ▁                                                             │
╰───────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯
╭─  ✓ CausalTransformer  ─────────────────────╮
│    trained  1 steps in 0:00:02              │
│ first step  after 1.39 s                    │
│    goodput  10.5% of the wall time in steps │
│ final loss  5.874                           │
╰─────────────────────────────────────────────╯

Recorded on a 4-vCPU Cloudflare container with JAX 0.11.2.post11 and Dew d2adda0 on 2026-10-21. The run took 35 s.

JAX/XProf capture, editable

Run runs the cell on a shared 4-vCPU host; training stops after 20 steps.

python profile.py
Wrote profile for 2 steps to profile
╭─  dew · CausalTransformer  ──────────────────────────────────────────────────────────────────╮
│ 18,560 parameters  ·  on 1 × cpu  ·  batch 8  ·  float32                                     │
│                                                                                              │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  5/5  100%  10.5 step/s  0:00:03 │
│ data 64 rows  ·  0 rejected                                                                  │
│                                                                                              │
│ train                                                                                        │
│   loss                  5.883  ▁                                                             │
│   ce                    5.883  ▁                                                             │
│   perplexity            358.9  ▁                                                             │
│   token_accuracy         0.7%  ▁                                                             │
│   data_wait_ms        0.04664  ▁                                                             │
│   grad_norm             1.758  ▁                                                             │
│   compiles              1.000  ▁                                                             │
│   cache_hits                0  ▁                                                             │
│   cache_misses          5.000  ▁                                                             │
│ throughput                                                                                   │
│   step_time_ms          113.2  ▁                                                             │
│   samples_per_sec       70.68  ▁                                                             │
│   tokens                500.2  ▁                                                             │
│   tokens_per_s          4,419  ▁                                                             │
│ optimizer                                                                                    │
│   accepted             100.0%  ▁                                                             │
│   learning_rate      0.001000  ▁                                                             │
╰───────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯
╭─  ✓ CausalTransformer  ────────────────────╮
│    trained  5 steps in 0:00:05             │
│ first step  after 4.56 s                   │
│       then  11.3 step/s                    │
│    goodput  7.2% of the wall time in steps │
│ final loss  5.883                          │
╰────────────────────────────────────────────╯

Recorded on a 4-vCPU Cloudflare container with JAX 0.11.2.post11 and Dew d2adda0 on 2026-10-21. The run took 35 s.

Checkpoint guarantees · Profiling

Try the 176M model

Tutorials

  1. 01Diffusion from scratchGPU
  2. 02Train a diffusion modelGPU
  3. 03Text to image with guidanceGPU
  4. 04Samplers and schedulesGPU
  5. 05Train a language modelGPU
  6. 06Representations with I-JEPAGPU
  7. 07Scale across devicesCPU
  8. 08Continue a pretrained decoderGPU
  9. 09SFT for LLMsGPU
  10. 10RL with verifiable rewardsGPU
  11. 11LLMs in an agent harnessGPU
  12. 12World modelsGPU
  13. 13Continual learning in contextGPU
  14. 14Vision-based drone racingGPU
How each example above was recorded
  • lm: a 4-vCPU Cloudflare container, JAX 0.11.2.post11, Dew d2adda0, 2026-10-21, 34.7 s.
  • jepa: a 4-vCPU Cloudflare container, JAX 0.11.2.post11, Dew d2adda0, 2026-10-21, 46.4 s.
  • finetune: a 4-vCPU Cloudflare container, JAX 0.11.2.post11, Dew d2adda0, 2026-10-21, 121.5 s.
  • decide: a 4-vCPU Cloudflare container, JAX 0.11.2.post3, Dew aee614a, 2026-10-08, 61.8 s.
  • diffusion: a 4-vCPU Cloudflare container, JAX 0.11.2.post11, Dew d2adda0, 2026-10-21, 36.9 s.
  • mesh: a 4-vCPU Cloudflare container, JAX 0.11.2.post11, Dew d2adda0, 2026-10-21, 34.5 s.
  • grpo: a 4-vCPU Cloudflare container, JAX 0.11.2.post16, Dew 8b4627a, 2026-10-24, 32.9 s.
  • reliability: a 4-vCPU Cloudflare container, JAX 0.11.2.post11, Dew d2adda0, 2026-10-21, 35.3 s.
  • profile: a 4-vCPU Cloudflare container, JAX 0.11.2.post11, Dew d2adda0, 2026-10-21, 34.7 s.
  • pretrained: a 4-vCPU Cloudflare container, JAX 0.11.2.post11, Dew d2adda0, 2026-10-21, 138 s.
  • serving: a 4-vCPU Cloudflare container, JAX 0.11.2.post3, Dew aee614a, 2026-10-08, 64.4 s.
  • formats: a 4-vCPU Cloudflare container, JAX 0.11.2.post4, Dew a758210, 2026-10-19, 79.1 s.