Building a High-Performance Elevation API

This article is reposted from Building a High-Performance Elevation API

At ColorfulClouds Technology(彩云科技， “cai yun” in Pinyin for 彩云), we are committed to providing users with meteorological data at higher temporal and spatial resolutions. Over the years, we have consistently faced a challenge: due to limitations in elevation data resolution, outdoor activity enthusiasts, particularly those involved in hiking and cross-country activities, often experience noticeable discrepancies between ColorfulClouds’ data and their actual experiences. This issue is especially prominent in mountainous and plateau regions with dramatic elevation changes.

To address this issue, we initiated an elevation data upgrade project in late 2023, aiming to provide users with more accurate meteorological information based on elevation data with a spatial resolution of 30 meters.

This article aims to introduce the implementation process of this new data system and share the challenges we encountered, our explorations, and solutions during the implementation.

Previous Elevation Data

Before early 2024, our elevation data was sourced from the SRTM (Shuttle Radar Topography Mission) dataset. This data was stored in PNG file format, where we determined pixel positions in the PNG by calculating the latitude and longitude of request points and computed elevation based on RGB values. Here’s a sample image showing elevation data of the Hengduan Mountains:

After internal processing, these data had a spatial resolution reduced to 5 kilometers and a vertical resolution of approximately 35 meters. Since we primarily focus on data for the China region, we could load this data directly into memory for calculations. The data retrieval latency was typically at the nanosecond level and therefore did not constitute a system bottleneck.

New Version of Elevation Data

Source

We directly downloaded the publicly available ASTER dataset from abroad, which is stored in GeoTIFF format with a spatial resolution of 30 meters and a vertical resolution of 1 meter. Due to the large data volume and need for preprocessing, we wrote the data directly to the JuiceFS distributed file system for convenient use in K8S.

Storage and Access

Initially, we converted the data from GeoTIFF format to .npy format for use in Python using the mmap approach for data reading.

However, we encountered several issues with this approach in practice:

1. The global data is approximately 500GB, but our actual business primarily focuses on the China region, which amounts to about 30GB. In K8S, services need to rely on disk mounting to start.
2. In high-concurrency scenarios, the Python+mmap approach couldn’t achieve the expected throughput.

To address the first issue, we attempted to use the JuiceFS distributed file system. Based on our previous experience, it can achieve read/write speeds comparable to mechanical hard drives and offers various deployment options in K8S. During testing, we found that under low traffic conditions, the service could achieve barely acceptable throughput. However, under high traffic, processes would continuously wait for file read results, causing Python services to freeze. Therefore, we made a trade-off by placing data for the China region on SSDs, as local disk latency is much lower than network IO. For specific clients requiring high-precision elevation data for overseas locations, we still keep overseas data on JuiceFS, which is acceptable given the current traffic pressure. Additionally, to address the issue of data file preparation during service startup, we configured the service as stateful to continue reusing the previous SSDs and added an init container for data downloading. The initialization process supports incremental downloads, resulting in very short total service startup times.

For the second issue, we attempted to implement elevation data reading in Go. We processed the original two-dimensional matrix into a one-dimensional int16 array, storing it in a binary file in a specific order. In the code, we calculate the offset in the file based on latitude, longitude, and spatial resolution, then use os.File.ReadAt to read data from specific positions:

1
2
3
4
5
6
7

data := make([]byte, dataSize)
_, err = f.ReadAt(data, offset)
if err != nil {
   return 0, err
}

value := binary.LittleEndian.Uint16(data)

As we can see, this method significantly reduced the file IO pressure. Through performance analysis using Pyroscope, we found that the main performance consumption was in the os.File.ReadAt for file reading and grid interpolation calculation phases. For the former, we expect to optimize through io_uring, which is a new asynchronous IO method introduced in Linux kernel version 5.1. Although K8S nodes currently support the io_uring mechanism, it requires privileged mode, which is currently unacceptable. Therefore, we focused our optimization efforts on grid interpolation calculations. Internal queries generate multiple large slices, causing significant performance overhead in memory allocation. By using sync.Pool to reuse slices, we achieved substantial performance improvements.

Currently, the service is configured with 1C+0.5GB memory, and under 1500 QPS pressure, the P99 access time is approximately 500 microseconds. Combined with the company’s current business scenarios, this latency level is acceptable.

The following screenshots show the web interface used for querying elevation data in the internal system:

Location	Screenshot
Jade Dragon Snow Mountain-Blue Moon Valley Scenic Area
Nanyue Mountain Weather Station (actual elevation ~1266m)
Mount Everest (actual elevation 8848.86m)
Giza Pyramids (actual elevation ~138m)
Nearby highlands where U.S. forces landed on Omaha Beach on D-Day

To verify the reliability of the data, we used trajectory data from the company’s team building trip to Lijiang in 2023, comparing the elevation data from the trajectory with our elevation data. The graph below shows the comparison between trajectory data and elevation data, demonstrating very small differences between the two.

Elevation Data	Trajectory Data

Conclusion

ColorfulClouds previously relied heavily on NumPy and mmap technology¹, which prevented us from realizing that many previous issues no longer exist today. By using more efficient programming languages combined with cheaper memory and SSDs, we can achieve better or almost consistent performance while significantly reducing service operating costs.

Of course, specific problems require specific analysis. For example, elevation data itself is relatively simple, with only one data point per location, making it easy to implement in Go, or even Rust.