In the previous post, 雪：更快速更流畅的气象数据动画 (Chinese),
I packed forecast variables into a single-file container, .xue, which the browser read frame by frame through its index.
For two variables of one GFS run,
per-frame precolored PMTiles (zoom 0–4) came to 3,205.87 MB,
the source GRIB2 was 137.73 MB,
and .xue brought it down to about 65 MB.
The size problem was solved. The problem with a custom container is that only its own decoder can read it.
On 2026-09-14 I posted the format to the Pangeo forum. The first reply, from Tom Nicholas, asked: is the binary format not what gives the performance, and all that is needed is visualization-friendly Zarr chunk shapes plus decoding in the browser with WASM?
Yes.
The speed comes from two things: quantizing to uint8 single-byte planes (4× smaller before compression, and a decoded plane uploads directly as a WebGL2 R8 texture), and chunks cut for playback.
Both can be expressed in standard Zarr v3.
Since 2026-09-15 .xue is no longer published: every run, every rolling-window round, and every historical case is published as Zarr v3 stores only, with a static STAC catalog as the entry point to the data.
The original .xue encoder and decoder code is kept as a reference for future work on the encoding.
Following the direction of data flow, the pipeline has these stages:
- Production: GitHub Actions fetches upstream data on a schedule, quantizes and encodes it into Zarr stores, and publishes them;
- Catalog: a static STAC catalog lists the current run of each source;
- Distribution: Cloudflare R2 stores the data, a CDN caches it, and clients read on demand over HTTP Range; an experimental read-only JSON API sits beside it;
- Rendering: Rust WebAssembly decodes in the browser, and WebGL2 projects and colors;
- Analysis: Python scripts read the stores directly, and the synoptic-analysis Skill produces synoptic analysis reports.
Data sources
| Source | Grid and cadence |
|---|---|
| NOAA GFS | 0.25°, hourly to F120, then 3-hourly to F240 |
| NOAA GFS Sflux | ~13 km Gaussian grid, same axis as GFS |
| ECMWF IFS open data | 0.25°, 3-hourly to F144, 6-hourly to F240 |
| ECMWF IFS open data 0.1° | 0.1°, same axis as above, about two hours after the 0.25° data |
| ECMWF AIFS Single | 0.25°, 6-hourly to F360 |
| ECMWF IFS HRES (via Open-Meteo) | 0.1°, hourly to F090, 3-hourly to F144, 6-hourly to F360; 00Z and 12Z |
| NOAA HRRR | 0.03° contiguous US, hourly to F18, a cycle every hour |
| DWD ICON-EU | 0.0625° Europe, hourly to F78, 3-hourly to F120 |
| Météo-France ARPEGE | 0.25°, hourly to F48, 3-hourly to F102 |
| NOAA GEFS-Aerosols | 0.25°, 3-hourly to F120 |
| CMA-CW/DUST dust forecast | 0.15° Asia, hourly to F72, 3-hourly to F168; 00Z and 12Z |
| NCEP CFSv2 | 0.9375° surface, 1° upper air, 6-hourly to F6552 (39 weeks); 00Z and 12Z |
| NOAA MRMS radar | 0.02° contiguous US, every 2 min, rolling 4 h |
| JMA precipitation nowcast | 0.005° Japan, every 5 min, rolling 3 h |
| Himawari-9 | 0.04°, every 10 min, rolling 6 h |
| Himawari-9 over Japan | 0.01°, 24–46°N 122–148°E, every 10 min, rolling 3 h |
| GOES-19 / GOES-18 | 0.04°, every 10 min, rolling 6 h |
| Meteosat-12 | 0.04°, hourly, rolling 24 h |
| NOAA SWPC aurora | 1°, every 5 min, rolling 12 h |
There is also a set of point products: soundings (one ascent per station), airports (METAR and TAF), tropical cyclones (per-storm tracks with agency and model forecasts), surface stations (AMeDAS, DWD, Hong Kong Observatory, aggregated every 10 minutes), air-quality stations (AirNow, Hong Kong EPD, every half hour), weather alerts (US NWS, JMA, ECCC, a snapshot every 5 minutes), and NASA FIRMS active fires (a rolling 7 days). Point products are JSON, fetched one line per station or storm with a Range request, and do not go through the raster format. NEXRAD single-site radar is the exception: the lowest sweep’s reflectivity and radial velocity have no lat/lon grid, so they are stored beam by gate as polar Zarr stores over a rolling 3 hours; the live view in the page reads NOAA’s Level 3 objects from the source bucket directly.
Only the newest run of each source stays online. Past weather events are published permanently as showcase cases, cropped to the event’s region and hours, with the same encoder and store format. Cases can also use sources that have no live feed: the ERA5 reanalysis, the CAMS aerosol analysis, EFAS river discharge, and a 500 m WRF nest over Mount Fuji.
Sources fall into two kinds. Forecast sources publish per cycle: one directory per run, replaced when a new run arrives. Observation sources (radar, geostationary satellites, and the aurora nowcast) are rolling windows: a round is built every few minutes, the window moves forward, and frames already fetched are not downloaded again. No source keeps one step all the way out, so the time axis declares its unit and the offset of every frame rather than assuming even spacing.
“Real-time” here means continuously updated forecasts and near-real-time observations: forecasts land 4–8 hours after the cycle, and the end of an observation window trails real time by a few minutes to half an hour, depending on upstream publication delay.
Differences between sources are absorbed at the fetch stage. Geostationary imagery is warped from the geostationary projection onto a lat/lon grid, IFS HRES is resampled from its reduced Gaussian grid, and HRRR from its Lambert conformal grid, each once on the fetch side; quantization, grouping, and store writing after that are the same path for every source.
Take MRMS: the product publishes a frame every two minutes, but a radar volume scan takes four to six minutes, so until a new low-elevation scan arrives, the frame inside that radar’s coverage repeats the previous one value for value. The visible result is that despite the high frame rate, echoes on the mosaic move in fits and starts: the echo areas share an overall direction, but in the details each part jumps on its own schedule.
Format and quantization
The access pattern is continuous playback of a whole global forecast run with free timeline scrubbing, and that constraint drives the format.
Every variable is quantized to one byte: 2 m temperature in 0.5 °C steps over −60 to 50 °C, with a maximum error of 0.25 °C; precipitation uses a logarithmic codebook up to 128 mm/h, keeping resolution in light rain. The quantized data keeps its numeric meaning, so the browser can color and play it, read the value at a grid point, and compare models.
The data stays on the model’s native grid, with no reprojection and no baked-in colors; projection and palette are left to the GPU.
In time, a chunk holds six frames. Frame differencing (the xue.delta codec) is on by default only in the series stores described below; the stores used for playback are not differenced, so a generic Zarr client needs no extra codec registered.
Precipitation is stored as is in every store, because precipitation areas move with weather systems and the frame differences are larger.
In space, the grid is cut into tiles whose size is set per source (52 × 48 for GFS and ECMWF), and the browser fetches only the tiles its viewport covers.
Layout
A bundle is the set of variables one layer uses; wind10m, for example, is the two variables ugrd10m and vgrd10m.
Since 10-23, every bundle of one resolution tier of a run is written into one Zarr group, the shared store:
bundles.zarr at full resolution, bundles.half.zarr with every second row and column, and for the geostationary disks also bundles.quarter.zarr with every fourth.
The group holds one uint8 array per variable, named by the variable id and shaped [frame, latitude, longitude];
the time, latitude, and longitude coordinate arrays are written once per store, and a bundle on a different axis or grid carries its own.
Each bundle’s metadata sits in the group document’s xue_bundles attribute.
An inner chunk is six frames by one tile, and each chunk is one independent Zstandard frame.
Each array is a single shard covering the whole time axis and the whole grid, so one array is two objects in the bucket (zarr.json and the shard), regardless of frame count.
The GFS run of 11-01 00Z has 79 bundles in its full tier, 93 variable arrays and 4 coordinate arrays: 195 objects with the group document.
According to the zarr-profile document, with one store per bundle a GFS run was 1,571 objects, of which only 190 were shards; most of the rest were the group document and coordinate arrays repeated in every store.
The shard index sits at the end of the object; the client fetches it with one HTTP suffix range, and every later frame and viewport is located through it.
The published codec chain is the standard [bytes, zstd], which any Zarr client reads without plugins.
Each store carries every bundle’s metadata and CF scale_factor / add_offset, so xarray.open_zarr returns physical values.
The other access pattern is one grid point’s whole series.
GFS tmp2m, prate, and wind10m, and a few surface variables on HRRR and sflux, are also published as series stores (<bundle>.series.zarr):
an inner chunk is the whole time axis by 8 × 8 grid points, with xue.delta on by default.
The zarr-profile document measured gfs.2026090918 (161 frames): the tmp2m series store is 37.2 MB against 37.0 MB for the playback store, and one grid point’s whole series is one chunk, 1.9 KB at the median.
A rolling window is rebuilt every few minutes, and two consecutive rounds differ only in their newest frames.
These sources’ stores are cut into segments by the clock: by the hour for feeds with frames less than an hour apart, by six hours (from 00, 06, 12, and 18 UTC) for hourly or slower ones.
Each segment is itself a shared store at <source>.segments/<segment start>/<digest>/bundles.zarr, the digest being the first eight hex digits of a SHA-256 over the segment’s metadata and codes.
A whole segment is byte-identical in every round that holds it, so a new round names the segments an earlier round published and writes only those whose contents changed, and a browser’s cached copies stay valid.
The Himawari-9 round starting at 01 UTC on 11-01 is made of six hourly segments.
Catalog
Each run writes a manifest.json, and beside it a derived STAC 1.1.0 catalog:
under the root Catalog, one Collection per source, one Item per run, one Item per round for the rolling windows, the same three levels for the point products, and one more Collection for the showcase with an Item per case.
The catalog is generated from the manifest and the source registry, with no timestamps or host names, so any document can be regenerated from the artifacts on disk; all links are relative.
The bucket keeps only the newest run of each source, so every Collection carries an item.json at a fixed path pointing to the current run.
Reading uses existing tools: pystac walks the catalog, and xpystac / odc-stac open an Item’s Zarr asset into xarray.
Every variable asset of a run’s Item points to the same bundles.zarr; the arrays in it are named by variable.
The group document carries inline consolidated metadata, so xr.open_zarr(url) works even on a plain HTTP bucket that cannot be listed.
Distribution
The site is a static page on Cloudflare Pages, the data sits in a public R2 bucket, a CDN sits in between, and the page reads the data without an API layer. R2 egress and edge hits are both free, so the number of requests mostly affects latency, not the bill.
The client reads the chunks it needs over HTTP Range, and adjacent ranges within one object are merged into one request. A global view costs one request per time chunk for a frame, a viewport one request per tile row, and a grid point’s whole series one chunk for a variable with a series store, otherwise one chunk per time chunk. Frames around the playhead are prefetched into a cache with a byte budget.
Before a new run goes live, the CDN is warmed first and the pointer switched afterwards, so the first visitor is served from the edge cache.
Computation is concentrated in the build; the page reads only static files. Spatial aggregation, stitching across runs, and similar work are precomputed into artifacts by the encoder.
There is also a read-only JSON data API at xue-api.ringsaturn.me, marked experimental, with no promise of compatibility, availability, or freshness.
It resolves a run along the STAC catalog’s path (Collection → pointer → manifest), reads the store with ranged requests, decodes with the same xue crate as the page’s WASM decoder, and answers in physical units:
/v1/point returns a series at a latitude and longitude, /v1/cell a grid cell’s series in a named run by row and column, and each point product has a station list and a per-station record.
The page does not depend on the API; when it is down, the site and the catalog keep working.
Rendering
In the browser, a Rust WebAssembly worker decodes chunks on demand,
and the WebGL2 fragment shader performs inverse Web Mercator projection per pixel and then looks up the palette.
Tiles are rectangles in grid space, so putting one back into the plane is a single texSubImage2D, and tiles that have not arrived are skipped in the shader.
Adjacent frames are blended on the GPU, so playback transitions smoothly.
Contour lines are drawn in the same shader without a second pass, and wind renders through a GPU particle layer.
The WebCodecs H.264 path (?use_h264=true) carries the same codes in losslessly encoded video. Since 10-29 no source builds that video, so the path applies only to older runs that still carry one.
The loop being played can be exported as a GIF or an MP4, the MP4 encoded in the browser by WebCodecs.
Point products are drawn as marks on the map: soundings, airports, surface stations, and active fires; air-quality stations are colored by PM2.5 concentration, and alerts fill their warning areas by level 1–5.
Analysis
The data is static files with open CORS and no keys, read without going through a wrapped API. The repository includes a synoptic-analysis Skill that, with an AI assistant able to run Python, reads real data from the STAC catalog, performs synoptic analysis, and writes a report. For example:
Analyze the synoptic situation over Tokyo for the next three days, compare the GFS and ECMWF forecasts of precipitation and wind, bring in the latest satellite, radar, and sounding data, and produce a report in English with figures, data times, and a discussion of uncertainty.
The Skill’s workflow is: fix the region and valid times, read the model fields, analyze upper-level troughs and ridges, jets, surface systems, moisture transport, and precipitation conditions,
check the models against observations, and compare where the models disagree.
The data for one analysis is saved locally first and then used for computation, plotting, and review, so the numbers and figures in a report come from the same data.
The region, focus, report language, and output format can all be changed.
When only the values at one place are needed, the data API’s /v1/point returns each variable’s series in physical units without running Python.
One report produced this way analyzes Typhoon Dujuan as it approaches Tokyo: Tokyo Typhoon Weather Process Analysis (PDF). How it was generated is described in Multi-Source Weather Analysis with LLMs.
Self-hosting
To run it independently, pick upstream open data and run the corresponding pipeline. One command builds a run locally:
| |
The dependencies are Python 3.14 or later (managed with uv), GDAL 3.8 or later with the GRIB driver, and Node.js 22 or later; building the WASM decoder needs the Rust toolchain and wasm-pack. Some sources also need eccodes or credentials.
The frontend’s data URL, R2 bucket, and paths are all configurable.
The data API’s code is in the repository as well and can be deployed separately.
Moving to another object store or an internal network requires changes to the publishing flow, network access, and permissions; the format, catalog, and decoder stay the same.
After deployment, what can be adjusted includes the data (start with one model and a few common variables, then extend as needed), the interface (palettes, layers, organizing content for a particular region), and the analysis (the Skill’s logic and report format).
The code is dual-licensed Apache 2.0 OR MIT, at your option; both permit commercial use, modification, and closed-source redistribution, requiring only that copyright and license notices be kept.
The specifications under docs/ (the container, the Zarr profile, the STAC catalog, the point products, the data API) are CC BY 4.0.
Published data keeps each source’s terms, which every STAC Collection names in its license and providers fields; Xue’s own contribution (the layout, the quantization, the catalog) is CC BY 4.0, credited by adding “via Xue (xue.ringsaturn.me)” after the source’s attribution.
Per source: NOAA products are in the public domain; ECMWF open data, the IFS HRES fields forwarded by Open-Meteo, DWD ICON-EU, and EUMETSAT’s hourly Meteosat data are CC BY 4.0 and require attribution; Météo-France ARPEGE is under the Licence Ouverte / Etalab 2.0; soundings and the CMA dust forecast are WMO core data under the Unified Data Policy; the JMA nowcast requires a source credit. The basemap is © OpenStreetMap contributors.
- Live demo: xue.ringsaturn.me
- Source code: github.com/ringsaturn/xue
- Data catalog: dataset.ringsaturn.me/xue