Better Python Project Maintenance

In the process of maintaining PyPI packages and projects, I often encounter two main pain points:

1. Complexity of Uploading and Publishing: In the past, uploading a PyPI package typically required configuring a PyPI Token, which poses a security risk. It also required local or CI packaging and uploading. For projects involving .whl files, additional environment configurations were necessary, which added to the workload.
2. Slow Dependency Locking: Larger Python projects often have many sub-dependencies, and not locking versions can lead to compatibility issues due to upstream changes. Traditional tools like PDM and Poetry are somewhat slow when locking dependencies, and the process can be frustrating. Back in 2021, I used Poetry to lock a project’s dependencies, and after 10 minutes, it still failed; since then, I haven’t used it.

Fortunately, solutions for these issues have emerged in recent years.

PyPI Trusted Publishers Support#

In 2023, PyPI announced support for a passwordless publishing mechanism called Trusted Publishers. This feature eliminates the need for a PyPI Token; you simply enter basic information about your GitHub Actions in the PyPI. The configuration process is straightforward, saving a lot of maintenance effort.

uv for Faster Dependency Locking#

In 2024, uv was released, greatly improving dependency locking speed for Python projects. Previously, tools like PDM and Poetry were useful, but dependency locking was relatively slow, especially in continuous integration. I set up a comparison repository on GitHub to showcase the dependency-locking speeds of various tools for a project I worked on (timing data based on GitHub Actions execution):

With uv, dependencies are locked in seconds, making the process almost unnoticeable during development.

Moreover, some Python projects depend on machine learning models, which inevitably require PyTorch and its ecosystem. This presents several challenges:

• Even with CPU-only versions, PyTorch tags differ across platforms, so versions must be selected according to the platform.
• Third-party projects in the PyTorch ecosystem are often strict about PyTorch version compatibility. When the installed version is incorrect, instead of clear error messages, you may encounter a cryptic Segmentation fault (core dumped).

To address this, I also use uv for such projects. By leveraging the index feature in uv, it’s easy to specify the source for PyTorch. For example, here’s a pyproject.toml file for a project using uv:

[project]
name = "foo-bar"
version = "0.1.0"
description = ""
readme = "README.md"
requires-python = "==3.12.*"

dependencies = [
    "torch==2.5.0; platform_system == 'Darwin'",
    "torch==2.5.0+cpu; platform_system != 'Darwin'",
]

[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cpu"
explicit = true

[tool.uv.sources]
torch = { index = "pytorch" }

Additionally, using tool.uv.find-links, it’s easy to configure sources for third-party projects, ensuring the correct versions are installed. This enables consistent environments across local development (macOS) and production (Linux), even for complex machine-learning projects. Dockerfiles are much cleaner too—no specific installation order required; simply run uv sync --compile --frozen.

Beyond that, uv can handle build, package, and publish tasks for Python packages and also supports PyPI’s Trusted Publishers, making maintenance much simpler. Here’s the GitHub Actions configuration file for the Apopy project to publish to PyPI:

name: publish

on:
  push:
    tags:
      - "v*.*.*"

jobs:
  pypi-publish:
    name: upload release to PyPI
    runs-on: ubuntu-latest
    environment:
      name: PyPI
      url: https://pypi.org/p/apopy
    permissions:
      id-token: write
    env:
      TRUSTED_PUBLISHING: always
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version-file: "pyproject.toml"
      - name: Install uv
        uses: astral-sh/setup-uv@v3
        with:
          enable-cache: true

      - run: make build
      - run: uv publish

Conclusion#

These methods have greatly simplified the maintenance of both company projects and my personal open-source projects, such as tzfpy, apopy, and cyeva.