WeaveRank Enriches Binder Hits 1.6-Fold: Validation on 1,880 Literature Designs

PFBSA: A Fluorinated Sulfonamide Target for Ligand-Conditioned Binder Design

A ligand-conditioned design campaign against perfluorobutanesulfonamide generated 195 binder candidates, with a top docking score of -7.74 kcal/mol and a target QED of 0.806.

Motivation

AffiniBind's rescore pipeline ends in a single number: WeaveRank, the score that decides which designed binders get ordered for synthesis. Candidates also carry a Pool score and per-channel metrics, but in the IVTT ordering lane, WeaveRank decides the order.

That makes one question load-bearing: does WeaveRank order candidates better than the alternatives, on designs it has never seen? The score was not so much designed as accumulated: an average of structural-confidence channels, with membership decided by a threshold. Answering honestly meant rebuilding the evidence at scale, splitting it by target, choosing on one side and reporting on the other.

An average over a whole ranking is not what a campaign spends money on. What it spends money on is the top of the ranking: the ten or twenty designs someone actually orders. So the headline number is the one a campaign lives by: sort by WeaveRank, take the top 10, and count how many are real binders.

One constraint shapes everything below: AffiniBind has no laboratory. Every quality claim here rests on literature-reported assay outcomes: binder calls made by other groups, with other assays, for other purposes. That is what this validation can measure, and it is not a wet-lab result.

What the Top 10 Buys

On the 1,880-design holdout, a median four of every ten designs at the top of a WeaveRank-sorted list are literature-confirmed binders. The median holdout target carries a 25% positive rate, so ten designs picked with no ranking at all would hold about two and a half binders; the ranked ten holds four: an enrichment of 1.6× (95% CI 1.23–1.80) over the target's own base rate. In the units a campaign actually spends, that is a hit every two designs tested, where a random order takes about three and a half.

Every column is a median over the targets where the metric is defined: for the holdout, 18 of the 24 targets clear the 20-row batch minimum, and 16 of the 18 put at least one binder in the top 10; for the selection read, all nine targets carry at least one positive. Enrichment at 10 is the precision of a target's top ten ranked designs divided by that target's positive rate; 1.0× is what an unranked list reads by construction. Tests per hit is ten divided by the hits in the top ten, medianed the same way and counted only on the targets where at least one hit exists; the random-order figure is the mean of that statistic over 200 draws of a permuted score, because a single draw of a noisy baseline is itself noise. Every headline number here carries an interval, and the intervals are wide: 24 targets is not many.

The two enrichment columns come from different phases, and the in-sample one is not equally trustworthy. The selection column is the same leave-one-target-out protocol run over the selection frame: 2,628 designs across the nine targets the choices were made on. The floor and the membership were chosen on that data, so the column is optimistic by construction: the shipped design reads 4.4× there. Most of the drop to 1.6× is not optimism but the metric's own ceiling moving: enrichment at 10 cannot exceed one over the target's positive rate, and the median selection target carries 9% positives against 25% on the holdout, a ceiling of 11.0× against 4.0×; the mechanism shows up at target level (Figure 1). What is stable across the two frames is the top-10 hit rate, 40% on both. The holdout column is the claim: 1.6× on 1,880 designs the design has never seen, targets excluded by sequence. The residual gap between the columns is the price of choosing on data, so the holdout number is the one this article leans on.

Two qualifications belong next to the number rather than at the end of the article. First, the baseline is real, but the margin is modest: a paired bootstrap over targets puts the enrichment gap against a random order at +0.60, with a 95% interval of +0.23 to +0.81. That interval is positive, but the effect stays a modest 1.6×. Second, the top 10 is the coarsest read in this article. The best single co-fold channel (iPTM) enriches 1.67× to the ensemble's 1.60×, and ten-design reads across 24 targets cannot separate a good channel from a good ensemble of channels. AP, the metric that membership and the floor are chosen on, is the finer instrument, and there the ensemble reads 0.512 against the single channel's 0.502, itself inside the noise. The ensemble ships because nothing beats it out of sample and it needs no fitted model.

One more distributional fact: 16 of the 18 holdout targets measurable at top-10 put at least one binder there, and the two that do not, human-rfk and trka, are named as weak spots below, not averaged away. Three further targets that carry positives (der21 16 rows, fgf-r1 18, human-phyh 12) fall under the 20-row batch minimum at this k and are unscored, so they are neither hits nor misses here.

What WeaveRank Is

WeaveRank is the mean of sign-aligned per-target z-scores of the admitted ESMFold2 co-fold interface-confidence channels, five at the current admission floor: ipSAE min (interface confidence derived from the co-fold interface PAE), iPTM (predicted interface TM-score), pLDDT avg (whole-complex confidence), LIS (a large-interface-size proxy) and interface PAE (mean predicted aligned error at the interface).

Each channel becomes a per-target z-score: centered and scaled against the target's own candidate batch, so a value means "confident relative to this target's other candidates", not "confident in absolute terms". Reference statistics fall back from the target's own batch (at least 20 rescored candidates) to a shipped per-target prior table, then to pooled global statistics.

Sign alignment is what makes a plain average meaningful: every member is flipped so higher always means more confident. Four channels enter with sign +1; interface PAE enters with sign −1, because for an error metric lower is better.

The admission floor decides membership: a channel joins the average only if its own single-feature median AP clears 0.40. At that floor the admitted set is exactly the five channels above. Both Protenix members sit below it, Protenix ipSAE min at 0.396 and Protenix iPTM avg at 0.337, so both are excluded and retained only as an audit trail. A candidate needs at least three present members to receive a score; below that the score is null and ordering falls back to the base score.

How It Was Validated

The Data: A 3,414-Design Corpus and a 1,880-Design Holdout

The inventory pass catalogued every labeled literature source in the repository: Overath 3,669 rows (3,414 rescorable), ProteinBase 1,511 (1,368), RFdiffusion releases 619 (96), and the Anthropic released set 1,440 rows, held off the pool because its targets need an inferred crosswalk, gated behind the --allow-inferred-crosswalks flag. That set is also this article's external benchmark: on corrected signs the shipped ensemble scores macro AP 0.522 (median 0.611) on the 1,320 labeled designs, using Anthropic's own shipped ef2fast/ef2full predictions, level with the ~0.52 Anthropic report themselves. The Bennett retrospective carries 603,178 labeled rows, but only about 2,500 of them come with a binder sequence. That is the difference between 610,417 rows and 4,783 foldable designs: a label with no design attached can never be scored.

Deduplication removed 501 rows shared with the smoke-test set and 3,057 duplicate (sequence, target) pairs. 25 duplicate rows disagreed with themselves: the same design called a binder by one assay and not another. The kept label wins, and the disagreement is recorded rather than averaged away. The final selection corpus is 3,414 rows, 344 positives, 15 target ids.

The holdout is a separate literature set whose targets appear nowhere in selection: 1,880 designs, 382 positives, 24 target ids over 21 distinct target sequences, each design carrying 38 columns, 34 of them rescore channels. Two unlabeled designs were dropped before any GPU work; nothing else was excluded. The holdout finished 1,880 of 1,880 folded, zero failed rows, zero exclusions, bringing total run spend, corpus and holdout together, to $128.38.

Data quality was audited, not assumed: since one target appears under several names in the source data (IL7Ra / il7r / il7ra), the split is drawn on the target sequence, never the name, a distinction that caught a real leak.

A Target-Disjoint Split

The protocol runs in two phases. A selection phase (nested leave-one-target-out over 9 targets, 2,628 rows, 232 positives) is where every choice is made. A holdout phase (the 1,880 designs, targets excluded from selection by sequence) is where the number is read. Selection is for choosing; only the holdout counts. Both enrichment phases are read through the same admission-gated mean-z path as the AP column below.

The Aggregations Compared

Convention A scores small batches through the production z fallback chain (per-target priors, then global stats) and medians over all 21 AP-computable targets; convention B leaves targets under the 20-row batch minimum unscored and medians over 18. The shipped row reads 0.512 under A and 0.508 under B, and the superseded six-member design reads 0.449 under B.

Two findings matter more than the ranking. First, the selection winner lost out of sample: XGBoost won selection at 0.525 and read 0.464 on the holdout, while the L1 variants gained. Acting on the selection number would have shipped the worse model. Second, no learned arm beat the unsupervised design: the best supervised read ties it at 0.512, and the shipped design is not the weakest arm on the holdout. L1-logistic weights over the full pool are not shipped because they need a fitted model; they carry the best pooled AUC of the five (0.727) but buy nothing out of sample the parameter-free ensemble does not already have. Nothing in the holdout argues for replacing the mean-z ensemble with a fitted model.

The Floor Change

The admission floor was the lever. A sweep over the corpus (mean-z over the admitted set) put the old floor of 0.35, six members including Protenix ipSAE min, at 0.425 selection median AP; floor 0.40, five members, reads 0.490, and 0.50 (four members) slips to 0.488. The holdout confirms the ordering: 0.449 for the six-member design against 0.508 for the five. The enrichment read agrees: the superseded six-member design enriches 1.23× where the shipped five-channel design enriches 1.6×.

The floor change removes exactly one member, and with the signs corrected the drop-one-member tests show no single removal from the admitted set to be a real re-admission signal. The deltas sit at the noise floor, between +0.018 and −0.030 holdout median AP, against per-target APs that span 0.06 to 0.90. The candidates that kept failing were the re-admission attempts in the other direction, bringing the Protenix channels back: three times, each losing out of sample. The floor was chosen on selection-phase evidence only; the holdout read is a confirmation, which is the difference between a validation and a rationalization.

The Sign Correction

The validation also found a bug. Both ipSAE min channels carried the wrong sign, annotated as inverted where the data want higher to mean more confident. The adjudication verified the inversion on both datasets: pooled Spearman +0.263 on the corpus, positive on 9 of 9 selection targets, and +0.376 on the smoke-test set, positive on 3 of 3. The old annotation described the underlying PAE, the error metric ipSAE is derived from, not ipSAE itself, which is a confidence transform. Exactly one of the two members is live at the 0.40 floor, and the drift check on AP was blind to the error: it measures AP with a sign-agnostic learner, so a flipped annotation reproduces the same numbers forever.

The previously published headline comparison, 0.513 for the five-member design against 0.417 for the old six-member design, predates that correction. Against the corrected member table, the same ablation convention gives 0.508 for the adopted five-channel design and 0.449 for the superseded six-member design; under the production fallback path the adopted design reads 0.512. The conclusion survives (the floor-0.40 ensemble still beats the old design), but the margin is smaller than the pre-correction pair implied: both configurations shift with the corrected annotation. The corrected-sign version is statistically indistinguishable from the shipped one: Δ +0.0205 on the production fallback convention, P(better) 0.515, a coin flip. It was adopted as a correctness fix, not a performance claim: it removes a false statement from the configuration and stops one admitted member contributing an inverted term to every production score. The convention-independent pooled AUC moves in the right direction, 0.6624 to 0.6752.

In the same round, two pre-registered candidates were rejected: a membership re-pin on re-measured APs (0.460) and a rank-normalized protocol variant (0.470), against the incumbent's 0.491 and 0.513 as measured in that same pre-correction read. Protenix membership has now failed to transfer out of sample three times.

The Production Path

The ensemble is a configuration, not a trained model. Membership, signs, and the floor are declared rather than learned, and the per-target reference statistics are regenerated from the 3,414-design corpus, most recently on 2026-09-11. Re-measuring the admission APs later is a one-file update. At scoring time the pipeline loads the configuration, resolves the admitted members, applies the reference fallback chain, sign-aligns every member, and emits the score plus a per-member z breakdown for observability.

The score is computed during the merge step that combines the per-target results, and that step emits the ranking score, its per-channel components, and the target grouping the z-scores are batched by; candidates without a score still get ranked, by the generator base score. The corrected configuration reached production too: a live campaign (28 candidates), never re-scored since the floor change, was run through the corrected path.

What This Means

1. The claim is enrichment in the top 10

Sort by WeaveRank, test the top ten, and a median four come back as literature-confirmed binders: 1.6× the target's base rate out of sample, a hit every two designs instead of roughly four, with 16 of the 18 targets measurable at top-10 returning at least one hit. Read on the selection frame, where the design was chosen, the same statistic is 4.4×; the 1.6× is the number that describes a new campaign, and most of the distance between the two is the metric's ceiling moving rather than optimism. Everything else in this article exists to make that number trustworthy and to say where it stops: it is a median over the 18 measurable literature targets, not a wet-lab rate, and the top 10 is too coarse a read to separate the ensemble from its best single channel.

2. What ships is simple

The adopted design has no fitted parameters: a fixed floor over measured admission APs, then a mean of sign-aligned z-scores. Out of sample it matched the best learned alternative and was not separable from its best single channel, with no fitted artifact and no new failure mode.

3. The split earned its keep

The selection-to-holdout split caught XGBoost's overfitting (0.525 to 0.464) before it could ship, and the same regression twice more on later proposals that looked better in selection.

4. Corrected numbers, smaller margin

The honest out-of-sample comparison is 0.508 against 0.449, not 0.513 against 0.417. The adopted design still wins on every convention tested, and it leads the enrichment table as well (1.6× against 1.23× for the design it replaced), but the AP margin is roughly 0.06 points, not 0.10, a margin worth carrying into any future claim about the score.

5. The weak spots are named, not explained away

20 of the 21 measurable holdout targets score above their own base rate. The one that does not, human-rfk, is the single below-chance target (0.064 against a 0.067 base rate), and it joins trka in returning no binder at all in the top 10. Three further targets carry zero positives, which made the "every target above chance" adoption clause unpassable as originally written; the clause was amended to targets with at least one positive and 20 rows, and the original verdict (reject) stands on the record alongside the adoption.

6. The holdout is spent

It has now been read for an adoption decision and may not be reused for future choices without becoming in-sample. The next lever is data, not aggregation: a ranked shortlist of external labeled binder datasets proposes a first wave of roughly 6,700 to 7,700 new rescored designs, recorded and deferred, not approved.

Data and Reproducibility

The validation is self-contained and archived in full: the run narrative, the evaluation output and adoption checks, the ablation results, the adjudication (pre-registration, sign verification, and the holdout read), and the corpus inventory with its per-chunk costs.

The enrichment figures are read from the holdout design matrix through the same admission-gated mean-z holdout path used for the AP columns above, together with the per-target metrics: enrichment at 10 and tests per hit, medianed over the 18 targets where enrichment at 10 is defined. The random-order baseline and the enrichment figures are recomputable from the scripts under experiments/weaverank/reports/figures_2026-09 (seed 42, 200 draws).

On the production side, the scoring path is a small set of configuration and code files, and the target-partition invariants are pinned by tests. Every number here traces to them.

See how the score sits inside the wider design pipeline, from fingerprinting through generation to developability ranking. View Platform

See the ranking score in candidate reports across open campaigns. View Open Targets