In the previous post, 雪：更快速更流畅的气象数据动画 (Chinese), I packed forecast variables into a single-file container, .xue, which the browser read frame by frame through its index. For two variables of one GFS run, per-frame precolored PMTiles (zoom 0–4) came to 3,205.87 MB, the source GRIB2 was 137.73 MB, and .xue brought it down to about 65 MB. The size problem was solved. The problem with a custom container is that only its own decoder can read it.

On 2026-09-14 I posted the format to the Pangeo forum. The first reply, from Tom Nicholas, asked: is the binary format not what gives the performance, and all that is needed is visualization-friendly Zarr chunk shapes plus decoding in the browser with WASM?

Yes. The speed comes from two things: quantizing to uint8 single-byte planes (4× smaller before compression, and a decoded plane uploads directly as a WebGL2 R8 texture), and chunks cut for playback. Both can be expressed in standard Zarr v3.

Since 2026-09-15 .xue is no longer published: every run, every rolling-window round, and every historical case is published as Zarr v3 stores only, with a static STAC catalog as the entry point to the data. The original .xue encoder and decoder code is kept as a reference for future work on the encoding.

Following the direction of data flow, the pipeline has these stages:

  1. Production: GitHub Actions fetches upstream data on a schedule, quantizes and encodes it into Zarr stores, and publishes them;
  2. Catalog: a static STAC catalog lists the current run of each source;
  3. Distribution: Cloudflare R2 stores the data, a CDN caches it, and clients read on demand over HTTP Range;
  4. Rendering: Rust WebAssembly decodes in the browser, and WebGL2 projects and colors;
  5. Analysis: Python scripts read the stores directly, and the synoptic-analysis Skill produces synoptic analysis reports.

Data sources

SourceGrid and cadence
NOAA GFS0.25°, hourly to F120, then 3-hourly to F240
NOAA GFS Sflux~13 km Gaussian grid, same axis as GFS
ECMWF IFS open data0.25°, 3-hourly to F144, 6-hourly to F240
ECMWF AIFS Single0.25°, 6-hourly to F360
ECMWF IFS HRES (via Open-Meteo)0.1°, hourly to F090, 3-hourly to F144, 6-hourly to F360; 00Z and 12Z
NOAA HRRR0.03° contiguous US, hourly to F18
NOAA GEFS-Aerosols0.25°, 3-hourly to F120
NCEP CFSv20.9375° surface, 1° upper air, 6-hourly to F6552 (39 weeks); 00Z and 12Z
NOAA MRMS radar0.02° contiguous US, every 2 min, rolling 4 h
JMA precipitation nowcast0.005° Japan, every 5 min, rolling 3 h
CMA radar mosaic0.0439° China, every 6 min, rolling 3 h
Himawari-90.04°, every 10 min, rolling 6 h
GOES-19 / GOES-180.04°, every 10 min, rolling 6 h
Meteosat-120.04°, hourly, rolling 24 h

There are also three point products: soundings (one ascent per station), airports (METAR and TAF), and tropical cyclones (per-storm tracks with agency and model forecasts). Point products are JSON, fetched one line per station or storm with a Range request, and do not go through the raster format.

Sources fall into two kinds. Forecast sources publish per cycle: one directory per run, replaced when a new run arrives. Observation sources (radar and geostationary satellites) are rolling windows: a round is built every few minutes, the window moves forward, and frames already fetched are not downloaded again. No source keeps one step all the way out, so the time axis declares its unit and the offset of every frame rather than assuming even spacing.

“Real-time” here means continuously updated forecasts and near-real-time observations: forecasts land 4–8 hours after the cycle, and the end of an observation window trails real time by a few minutes to half an hour, depending on upstream publication delay.

Differences between sources are absorbed at the fetch stage. Geostationary imagery is warped from the geostationary projection onto a lat/lon grid, IFS HRES is resampled from its reduced Gaussian grid, and HRRR from its Lambert conformal grid, each once on the fetch side; quantization, grouping, and store writing after that are the same path for every source.

Take MRMS: the product publishes a frame every two minutes, but a radar volume scan takes four to six minutes, so until a new low-elevation scan arrives, the frame inside that radar’s coverage repeats the previous one value for value. The visible result is that despite the high frame rate, echoes on the mosaic move in fits and starts: the echo areas share an overall direction, but in the details each part jumps on its own schedule.

Format and quantization

The access pattern is continuous playback of a whole global forecast run with free timeline scrubbing, and that constraint drives the format.

Every variable is quantized to one byte: 2 m temperature in 0.5 °C steps over −60 to 50 °C, with a maximum error of 0.25 °C; precipitation uses a logarithmic codebook up to 128 mm/h, keeping resolution in light rain. The quantized data keeps its numeric meaning, so the browser can color and play it, read the value at a grid point, and compare models.

The data stays on the model’s native grid, with no reprojection and no baked-in colors; projection and palette are left to the GPU. In time, frames go in groups of six: smooth fields are differenced frame to frame within a group, while precipitation is stored as is, because precipitation areas move with weather systems and the frame differences are larger. In space, the grid is cut into tiles whose size is set per source (52 × 48 for GFS and ECMWF), and the browser fetches only the tiles its viewport covers.

Layout

A bundle is one Zarr group holding one uint8 array per variable, shaped [frame, latitude, longitude], plus three coordinate arrays for time, latitude, and longitude. An inner chunk is six frames by one tile, and each chunk is one independent Zstandard frame.

Each array is a single shard covering the whole time axis and the whole grid, so one array is one object in the bucket. A scalar store is 9 objects regardless of frame count, and one GFS run is about 1,200 objects. The shard index sits at the end of the object; the client fetches it with one HTTP suffix range, and every later frame and viewport is located through it.

The published codec chain is the standard [bytes, zstd], which any Zarr client reads without plugins. Each store carries the bundle metadata and CF scale_factor / add_offset, so xarray.open_zarr returns physical values.

Catalog

Each run writes a manifest.json, and beside it a derived STAC 1.1.0 catalog: under the root Catalog, one Collection per source, one Item per run, one Item per round for the rolling windows, and the same three levels for the point products. The catalog is generated from the manifest and the source registry, with no timestamps or host names, so any document can be regenerated from the artifacts on disk; all links are relative.

The bucket keeps only the newest run of each source, so every Collection carries an item.json at a fixed path pointing to the current run.

Reading uses existing tools: pystac walks the catalog, and xpystac / odc-stac open an Item’s Zarr asset into xarray. The group document carries inline consolidated metadata, so xr.open_zarr(url) works even on a plain HTTP bucket that cannot be listed.

Distribution

The site is a static page on Cloudflare Pages, the data sits in a public R2 bucket, a CDN sits in between, and there is no API layer. R2 egress and edge hits are both free, so the number of requests mostly affects latency, not the bill.

The client reads the chunks it needs over HTTP Range, and adjacent ranges within one object are merged into one request. A global view costs one request per time chunk for a frame, a viewport one request per tile row, and a grid point’s whole series one chunk per time chunk. Frames around the playhead are prefetched into a cache with a byte budget.

Before a new run goes live, the CDN is warmed first and the pointer switched afterwards, so the first visitor is served from the edge cache.

Computation is concentrated in the build; online there are only static files. Spatial aggregation, stitching across runs, and similar work are precomputed into artifacts by the encoder.

Rendering

In the browser, a Rust WebAssembly worker decodes chunks on demand, and the WebGL2 fragment shader performs inverse Web Mercator projection per pixel and then looks up the palette. Tiles are rectangles in grid space, so putting one back into the plane is a single texSubImage2D, and tiles that have not arrived are skipped in the shader. Adjacent frames are blended on the GPU, so playback transitions smoothly.

Contour lines are drawn in the same shader without a second pass, and wind renders through a GPU particle layer. An optional WebCodecs H.264 path (?use_h264=true) carries the same codes in losslessly encoded video, with smaller files.

Analysis

The data is static files with open CORS and no keys, read without going through a wrapped API. The repository includes a synoptic-analysis Skill that, with an AI assistant able to run Python, reads real data from the STAC catalog, performs synoptic analysis, and writes a report. For example:

Analyze the synoptic situation over Tokyo for the next three days, compare the GFS and ECMWF forecasts of precipitation and wind, bring in the latest satellite, radar, and sounding data, and produce a report in English with figures, data times, and a discussion of uncertainty.

The Skill’s workflow is: fix the region and valid times, read the model fields, analyze upper-level troughs and ridges, jets, surface systems, moisture transport, and precipitation conditions, check the models against observations, and compare where the models disagree. The data for one analysis is saved locally first and then used for computation, plotting, and review, so the numbers and figures in a report come from the same data. The region, focus, report language, and output format can all be changed.

One report produced this way analyzes Typhoon Dujuan as it approaches Tokyo: Tokyo Typhoon Weather Process Analysis (PDF). How it was generated is described in Multi-Source Weather Analysis with LLMs.

Self-hosting

To run it independently, pick upstream open data and run the corresponding pipeline. One command builds a run locally:

1
2

make mvp MODEL=gfs    # or ecmwf / aifs / ifshres / sflux / hrrr / gefsaero / cfs
make serve

The dependencies are Python, GDAL with the GRIB driver, and Node.js; building the WASM decoder needs the Rust toolchain. The frontend’s data URL, R2 bucket, and paths are all configurable. Moving to another object store or an internal network requires changes to the publishing flow, network access, and permissions; the format, catalog, and decoder stay the same.

After deployment, what can be adjusted includes the data (start with one model and a few common variables, then extend as needed), the interface (palettes, layers, organizing content for a particular region), and the analysis (the Skill’s logic and report format).

The code is dual-licensed Apache 2.0 OR MIT, at your option; both permit commercial use, modification, and closed-source redistribution, requiring only that copyright and license notices be kept. Data licenses differ by source: NOAA products are in the public domain; ECMWF open data and the IFS HRES fields forwarded by Open-Meteo are CC BY 4.0 and require attribution; the basemap is © OpenStreetMap contributors.