ringsaturn's blog

About

Posted on Jan 1, 0001

Work Experience

I have been working with Beijing ColorfulClouds Technology Co.,Ltd as a Backend Engineer since July 2019. Here, I am responsible for:

  • Upgrading our Weather Core API from Python 2.7 to Python 3.7 in 2019.
    • A transition was made from maintaining a forked ujson library, nujson, to deploying orjson in the third quarter of 2022.
  • Refactoring the push notification system for the ColorfulClouds app.
    • This involved using a Python Producer with Celery workers to write push tasks as Protocol Buffers to a Redis list, which were consumed by a Golang worker.
    • This approach enabled us to send about 3,000 notifications per second.
  • Recoding high throughput internal Flask & FastAPI services using Go+Gin to offer high speed geo computing and weather station data APIs.
    • This allowed us to reduce infrastructure to 4 machines (previously 7), each with 8 CPUs and 16GB memory, while achieving a maximum QPS of 10K and a p99 latency under 2ms.
    • As a result, the Core API’s response time was shortened to 20ms when calling these internal services.
    • The new system can compute input (latitude, longitude) data to identify its administrative area in less than 0.02ms through a polygon search.
  • Revamping many years of legacy code to fit into modern architecture under Kubernetes.
    • We managed to achieve cross region traffic migration without any downtime in Q1 2023.
  • Developing an animation system for weather layer, demo, which has proven hugely beneficial for marketing purposes.
  • Creating a system to archive and query historical weather station data. The system stores data in multiple Parquet files and utilizes a few Celery workers to perform asynchronous read and processing tasks.

Technologies I used:

  • Redis (PubSub / Hash / List)
  • MongoDB (Geo Index / Aggregation)
  • Kubernetes (Deployment / Statefulset / Cronjob)
  • Python (Flask / Gunicorn / Uwsgi / NumPy / FastAPI / Celery / Parquet / Polars)
    • We decided to stop using FastAPI for CPU intensive services as Python’s AsyncIO scheduler wasn’t efficient for high throughput QPS needing multiple cores for acceleration.
    • However, we continue to use Flask services requiring specific features that only NumPy can provide, which isn’t possible with Go.
  • Geo Computing (H3 / Slippy Tiles / Timezone / Administrative Boundary Check / Point-In-Polygon / Ray casting / …)
  • Go: (Gin / Protocol Buffers / gRPC)
  • Rust (Creating Python extensions, such as tzfpy)

Side Projects

I have developed several side projects:

  • Fast timezone packages
  • A forked version of wego with Caiyun API support, demo
  • Go package aqi for computing AQI (Air Quality Index)

Education

I obtained a bachelor’s degree in Information and Computing Science from the North China University of Technology from 2015 to 2019.

More Info