BlackTree Security · Infrastructure · Automation · AI

LiteLLM’s New Signing Key Does Not Clean Old Deployments

LiteLLM has added a new signing-key fingerprint to its account of October’s compromised release. That helps operators check the software they intend to deploy. It does not clean an environment that already ran the malicious package, end a surviving process or revoke credentials the process could reach.

The distinction matters because the poisoned container images themselves carried a valid signature from the previous key. A signature answers a question about the signer and artifact. It cannot make an attacker-controlled release trustworthy when the publishing authority behind that signature has been compromised.

What changed on 30 October

The maintainer’s incident notice now includes a 30 October update: the commit that rotated the image-signing key on 23 October is shown as verified on GitHub, and the new public-key fingerprint has been added. The notice says the first releases signed with that replacement key were v1.103.8, v1.104.6, v1.105.4 and v1.106.3, published on 24 October. The dates matter here; a numerically lower maintenance release can have been published after the malicious v1.107.0.

The current image-security guide identifies the replacement key and its fingerprint, describes verification against a pinned repository commit, and explains digest pinning. It limits trust in the old key to images published before the compromised release on 21 October. Check the actual key, registry, image digest and publication history together. A passing check against whichever key happens to be cached locally is an incomplete recovery record.

Keep the registry windows separate

The affected release was distributed on 21 October. The maintainer’s current notice distinguishes these artifact and withdrawal windows, all in UTC:

Distribution routeAffected scopeRecorded availability
PyPIAll seven LiteLLM 1.107.0 wheels11:39:51 until removal by 13:23
Docker Hubv1.107.0, latest, main-latest and main-stableFrom 11:46:56 until removal or repointing at 14:20
GHCRThe affected core, database and non-root image indexes, across the listed release and rolling tagsFrom 11:46:56 until unavailable by 13:51

Use the notice’s complete affected-digest list to check retained images, mirrors and build records. A floating tag can now point somewhere different from the image originally pulled. The official GitHub source tag does not contain the malicious files, so finding a clean source checkout does not establish that a downloaded wheel or container was clean. The maintainer excludes its private enterprise registry from this release incident; that is a specific distribution boundary, not a general security guarantee for every enterprise deployment.

The later investigation corrected the first explanation

LiteLLM’s 23 October investigation supersedes the initial theory that developer tokens were obtained from application response logs. Its account starts with an administrative key exposed in a public repository on 27 May. Deleting the file did not revoke the key. The company says it remained usable and allowed access to an internal Agent Platform deployment on 10 October, from which an organisation owner’s GitHub CLI OAuth token was retrieved.

The reported chain then crossed from that identity into repository workflows and additional credentials. The attacker changed the release source to a lookalike repository and used a separate stolen maintainer identity to approve publication. An owner-level credential could override branch protection. Multiple approval steps therefore did not supply independent protection once the identities controlling them were compromised.

That is the maintainer’s upstream investigation. Microsoft’s separate technical analysis does not independently establish the initial access to the release environment. Keep those evidence scopes distinct. This incident is also separate from the previously repaired gateway permission flaws described in BlackTree’s earlier LiteLLM report.

Do not launch Python just to check the installation

Microsoft’s analysis explains why exposure was wider than an application importing LiteLLM. The affected wheels placed startup components outside the ordinary application directory. Normal interpreter initialisation could activate them, including when Python was started to install an unrelated dependency. Starting Python or pip inside a suspect environment can therefore trigger the behaviour an investigator is trying to examine.

Microsoft distinguishes the credential collector from a separate Unix-oriented loader capable of retrieving further code. It reports endpoint observations consistent with collector behaviour on Windows and macOS, while additional-code retrieval and execution through the loader were not confirmed. A collection capability is not proof that every targeted secret was obtained. Equally, an absent temporary marker is not proof of a clean host.

Close three different recovery questions

BlackTree recommends keeping artifact acceptance, environment recovery and identity recovery as separate decisions. First, establish which exact artifact ran and which verified replacement will be admitted. Second, investigate the affected environment and rebuild it from trusted material rather than assuming removal of the application directory removed every startup component. Third, revoke or rotate reachable credentials and review active sessions from a known-clean machine.

Scope that work to the affected identity’s real access. A developer workstation, build agent, container and gateway database expose different resources. Preserve the artifact reference, execution window, relevant process and network evidence, credential owners and outstanding checks. Record an unknown as an unknown instead of turning a successful replacement deployment into a claim about earlier access.

The new signing key is useful evidence for the next deployment. Recovery from the previous one still needs its own evidence. BlackTree has reviewed the public notices and analysis; it has not executed the affected package, tested a reader’s environment or verified any organisation’s incident closure.

Leave a Reply

Your email address will not be published. Required fields are marked *