Audit Who Writes Your Agent’s Rules
PackHallu preprint.
Published 7 October 2026, PackHallu used a local PyPI mirror, harmless marker and disabled manual confirmations.
The defensive question begins before installation. Who supplied the instructions that the agent treats as project policy, and who is allowed to change a dependency name, source or lockfile?
A rule file may look like documentation, but an agent can use it to shape code and tool decisions. Once the workflow can resolve and install packages, prose has reached a software supply-chain boundary.
The percentage needs a denominator
Five-model OpenHands BigCodeBench averages: 79.29% syntactic, 67.68% deployable, not eight-framework, thirteen-model or production-task rates.
Syntactic eligibility used benign intended-package imports; deployable added successful benign execution.
Deployable fell to 45.00% on DS-1000 and 16.28% on RefactorBench.
The decline matters. A strong result in a file-level coding benchmark does not become a universal success rate for complex repositories, every agent stack or supervised work. The test shows a credible mechanism and where it weakened.
A detector cannot make the approval decision
Same-author detector data comprised 100 benign and 100 malicious rules, not an independent broad benchmark.
That makes the detector results useful as a warning rather than a procurement scorecard. A scanner can prioritise review, but it should not be able to approve a new dependency by itself. The final gate should check a concrete package name, registry, publisher and resolved version.
Put controls after the prompt
BlackTree recommends:
- Version the rules. Keep agent instruction files under the same review and ownership controls as build and workflow configuration.
- Approve dependency changes. Require a person to review new package names, registry changes and lockfile differences before installation.
- Constrain resolution. Use approved registries, namespace policy and package provenance checks that the rule file cannot override.
- Reduce execution reach. Run untrusted coding work without production credentials and with limited filesystem access and outbound traffic.
- Join the audit trail. Correlate rule-file changes with dependency-manifest changes and package installation events.
The important boundary is deterministic. Agent reasoning may help explain a change, but identity, provenance and policy should decide whether the selected package can enter the environment.
What the study does not establish
BlackTree has not reproduced the preprint, and this review established no independent reproduction.
No production exploit campaign or real malicious package is documented. Success requires an adopted poisoned rule and available chosen package. No BlackTree control was tested.
Those limits keep the conclusion useful. Teams have evidence for a trust-boundary review without treating a controlled benchmark as proof of current compromise.
