PyPI in 2025: A Year in Review

1/1/2026 at 9:12:54 AM

> Trusted Publishing

Why do people come up with such unbelievably complex solutions that don’t actually achieve what a simple solution could do?

Trusted Publishing approximately involves a service like GitHub proving to somebody that some release artifact came from a GitHub Actions workflow file with a particular name, possibly in a particular commit. Never mind that GitHub Actions is an unbelievable security nightmare and that it’s probably not particularly hard for a malicious holder of GitHub credentials to stealthily or even completely silently compromise their own Actions workflow to produce malicious output.

But even ignoring that, it’s wildly unclear what is “trusted”. PyPI encourages developers to also use “attestations”. Read this and try to tell me what is being attested to:

https://docs.pypi.org/attestations/producing-attestations/

But I did learn that this is based on Sigstore. Sigstore is very impressive: it’s a system by which GitHub can attest via OIDC to various state, and a service called Fulcio (which we’re supposed to trust) uses its secret key to sign a message stating that GitHub did so at a certain time. (The OIDC transcript itself is not a durable attestation.) There’s even a transparency log (which is a separate system called Rekor maintained by the same organization). Except that, for some reason, Fulcio doesn’t do that at all. Instead it issues an X.509 certificate with an expiration in the near future where the certificate fields encode whatever GitHub attested to in its OIDC exchange, and the Sigstore client (which is hopefully a bit trustworthy) is supposed to use the private key (which it knows, in the clear, but is supposed to immediate forget) to sign a message that is associated with the release artifact or whatever else is being attested to. And then a separate transparency log records the signature and supposedly timestamps it so everyone one can verify the attestation later even though the certificate is expired! Why not just sign the message on the Fulcio server (which has an HSM, hopefully) directly?

All of this is trying to cryptographically tie a package on PyPI.org to a git tag. But: why not just do it directly? For most pure Python packages, which is a whole lot of packages, the distribution artifact is literally a zip file containing files from git, verbatim, plus some metadata. PyPI could check the GitHub immutable tag, read the commit hash, and verify the whole chain of hashes from the files to the tree to the commit. Or PyPI could even run the build process itself in a sandbox. (If people care about .pyc files, PyPI could regenerate them (again, in a sandbox), but omitting them might make sense too — after all, uv doesn’t even build them by default.) This would give much stronger security properties with a much more comprehensible system and no dependence on the rather awful security properties of GitHub Actions.

by amluto

1/1/2026 at 11:55:04 AM

Golang did the right thing by just wrapping git clone.

by never_inline

12/31/2025 at 11:37:30 PM

One of the big companies making billions on Python software should step up and fund the infrastructure needed to enable PyPI package search via the CLI, like you could with `pip search` in the past.

by heavyset_go

1/1/2026 at 12:37:06 AM

Serious question: how important is `pip search` to your workflows? I don’t think I ever used it, back when PyPI still had an XMLRPC search endpoint.

(I think the biggest blocker on CLI search isn’t infrastructure, but that there’s no clear agreement on the value of CLI search without a clear scope of what that search would do. Just listing matches over the package names would be less useful than structured metadata search for example, but the latter makes a lot of assumptions about the availability of structured metadata!)

by woodruffw

1/1/2026 at 12:41:09 PM

Not important at all now, given that it hasn't worked in a decade and I've filed it away as pointless to even consider for a workflow.

However, I get a lot of mileage out of package repository search with package managers like pacman, apt, brew, winget, chocolatey and npm.

> I think the biggest blocker on CLI search isn’t infrastructure

It's why it was shut down, the API was getting hammered and it cost too much to run at a reasonable speed and implement rate limiting or whatever.

by heavyset_go

1/1/2026 at 3:39:47 PM

> It's why it was shut down, the API was getting hammered and it cost too much to run at a reasonable speed and implement rate limiting or whatever.

Sort of: the original search API used a POST and was structured with XML-RPC. PyPI’s operators went to great efforts to scale it, but that wasn’t a great starting point. A search API designed around caching (like the one used on PyPI’s web UI) wouldn’t have those problems.

by woodruffw

1/1/2026 at 3:10:05 AM

I upvoted you because I broadly agree with you, but search is never coming back in the API. They previously outlined the cost involved and there's no way, given how minimal the value it gives more broadly, it's coming back ant time soon. It's basically an abusive vector because of the compute cost.

by talideon

1/1/2026 at 12:08:21 AM

Funding could help, but it still requires PyPI/Warehouse to ship and operate a new public search interface that is safe at internet scale.

by firesteelrain

1/1/2026 at 12:51:25 AM

They operate a public package hosting interface, how is a search one any harder?

by coldtea

1/1/2026 at 1:36:10 AM

PyPI responses are cached at 99% or higher, with less infrastructure to run.

Search is an unbounded context and does not lend itself to caching very well, as every search can contain anything

by miketheman

1/1/2026 at 2:02:42 AM

Pypi has fewer than one million projects. The searchable content for each package is what? 300 bytes? That's a 200mb index. You don't even need fancy full text search, you could literally split the query by word and do a grep over a text file. No need for elasticsearch or anything fancy.

And anyway, hit rates are going to be pretty good. You're not taking arbitrary queries, the domain is pretty narrow. Half the queries are going to be for requests, pytorch, numpy, httpx, and the other usual suspects.

by bastawhiz

1/1/2026 at 4:03:12 AM

I wonder how a PyPi search index could be statically served and locally evaluated on `pip search`?

by froh

1/1/2026 at 4:25:20 AM

PyPI servers would have to be constantly rebuilding a central index and making it available for download. Seems inefficient

by firesteelrain

1/1/2026 at 4:15:35 PM

Debian is somehow able to manage it for apt.

by ptx

1/1/2026 at 5:17:53 PM

1. Debian is local first via client side cache

2. apt repositories are cryptographically signed, centrally controlled, and legally accountable.

3. apt search is understood to be approximate, distro-scoped, and slow-moving. Results change slowly and rarely break scripts. PyPI search rankings change frequently by necessity

4. Turning PyPI search into an apt-like experience would require distributing a signed, periodically refreshed global metadata corpus to every client. At PyPI’s scale, that is nontrivial in bandwidth, storage, and governance terms

5. apt search works because the repository is curated, finite, and opinionated

by firesteelrain

1/1/2026 at 6:52:09 PM

isn't this an incrementally updatable tree that is managed with a Merkle tree? git-like, essentially?

by froh

1/1/2026 at 7:21:22 PM

The install side is basically Merkle-friendly (immutable artifacts, append-only metadata, hashes, mirrors). Search isn’t. Search results are derived, subjective, and frequently rewritten (ranking tweaks, spam/malware takedowns, popularity signals). That’s more like constantly rebasing than appending commits.

You can Merklize “what files exist”; you can’t realistically Merklize “what should rank for this query today” without freezing semantics and turning CLI search into a hard API contract.

by firesteelrain

1/1/2026 at 6:48:05 PM

that depends on how it can be downloaded incrementally.

by froh

1/1/2026 at 4:55:07 AM

The searchable context for a distribution on PyPI is unbounded in the general case, assuming the goal is to allow search over READMEs, distribution metadata, etc.

(Which isn’t to say I disagree with you about scale not being the main issue, just to offer some nuance. Another piece of nuance is the fact that distributions are the source of metadata but users think in terms of projects/releases.)

by woodruffw

1/1/2026 at 3:34:52 PM

> assuming the goal is to allow search over READMEs, distribution metadata, etc.

Why would you build a dedicated tool for this instead of just using a search engine? If I'm looking for a specific keyword in some project's very long README I'm searching kagi, not npm.

I'd expect that the most you should be indexing is the data in the project metadata (setup.py). That could be unbounded but I can't think of a compelling reason not to truncate it beyond a reasonable length.

by bastawhiz

1/1/2026 at 3:43:15 PM

You would definitely use a search engine. I was just responding to a specific design constraint.

(Note PyPI can’t index metadata from a `setup.py` however, since that would involve running arbitrary code. PyPI needs to be given structured metadata, and not all distributions provide that.)

by woodruffw

1/1/2026 at 5:42:48 PM

>The searchable context for a distribution on PyPI is unbounded in the general case, assuming the goal is to allow search over READMEs, distribution metadata, etc.

Even including those, it's what? Sub-20-30GB.

by coldtea

1/1/2026 at 10:42:38 AM

How does the big white search box at https://pypi.org/ work? Why couldn’t the same technology be used to power the CLI? If there’s an issue with abuse, I don’t think many people would mind rate limiting or mandatory authentication before search can be used.

by Kwpolska

1/1/2026 at 1:42:35 PM

The PyPI website search is implemented using a real search backend (historically Elasticsearch/OpenSearch–style infrastructure) layered behind application logic on Python Package Index. Queries are tokenized, ranked, filtered, logged, and throttled. That works fine for humans interacting through a browser.

The moment you expose that same service to a ubiquitous CLI like pip, the workload changes qualitatively.

PyPI has the /simple endpoint that the CDN can handle.

It’s PyPI philosophy that search happens on the website and pip has aligned to that. Pip doesn’t want to make a web scraper understandably so the function of searching remains disabled

by firesteelrain

1/1/2026 at 2:03:12 AM

Pypi has a search interface on their public website, though?

by bastawhiz

1/1/2026 at 8:49:48 AM

If you really need it, they publish a dump regularly and you can query that.

For simple use cases, you have the web search, and you can curl it.

by BiteCode_dev

1/1/2026 at 2:25:54 AM

They probably don't need it. You can start a crowdfunding campaign if you do.

by rat9988

1/2/2026 at 3:26:05 AM

> More than 3.9 million new files published

> More than 130,000 new projects created

Is there any way to prevent PyPI from becoming a morass of supply chain attacks like NPM etc.? The cited security measures (though some of them like domain resurrection protection are probably very good ideas) seem like they won't, but it also seems like a very hard problem to solve, given the vast scale as well as core issues like malicious (but seemingly innocuous) upstream commits.

by musicale

12/31/2025 at 11:11:45 PM

Great work Dustin and team!

by dalanmiller

1/1/2026 at 1:12:05 AM

Great work!

Side issue: anyone else seeing that none of the links in the article work? They're all 404s.

by nmstoker

1/1/2026 at 1:50:20 AM

Whoops, sorry about that. Should be fixed now. Happy New Year!

by miketheman

1/1/2026 at 10:22:22 AM

Thanks! I confirm they're all working for me now!

Happy New Year!

by nmstoker

1/1/2026 at 5:54:02 AM

> 1.92 exabytes of total data transferred

That's something like triple the amount from 2023, yes?

by zahlman

1/1/2026 at 7:51:11 AM

Is the compute and network required to service pypi all from donations or do they have any business arm that generates income?

by nodesocket

1/1/2026 at 2:38:00 AM

This seems to suggest once the bubble pops, it will take Python down with it. The next AI winter will definitely replace Lisp with Python.

by fud101

1/1/2026 at 10:13:05 AM

Replace lisp with python?

Edit: my bad it seems you meant the opposite. Absolutely fantasy but a man can certainly dream lol

by chamomeal

1/1/2026 at 3:10:35 AM

Appropriate username!

by talideon