Redocking and docking validation: the 2 Å RMSD rule and PoseBusters
What a passed redocking test proves, and what it leaves open
Matic Broz, PhD19 min

Redocking and docking validation
Redocking means taking a crystal structure of a protein with its ligand, removing the ligand, docking it back into the same protein and checking whether the top-ranked pose lands within 2 Å RMSD of the crystal pose.
It is the first validation step because it is the easiest question a docking setup can face and the only one with an experimental answer. If the receptor preparation, search box and docking program cannot rebuild a pose that the pocket was crystallized with, they will not do better on a molecule nobody has crystallized. Success rates are high: with relatively unbiased protocols, GOLD redocked about 80% of the Astex test set correctly.[6]
Passing proves less than it appears to. We redocked imatinib into the c-Abl kinase structure 1IEP with AutoDock Vina and got a top pose 0.75 Å from the crystal pose. Docked into the nilotinib-bound structure 3CS9, the top pose was 1.26 Å away. Docked into the dasatinib-bound structure 2GQG, it was 10.03 Å away, scoring about 3 kcal/mol worse. All 19 poses from the four runs, three correct and 16 wrong, passed every chemistry, geometry and clash test in PoseBusters.
A complete validation asks four questions. Does the setup reproduce known poses? Does it still do so in a receptor structure built around a different ligand? Does it separate known binders from decoys? Are the poses physically possible? Redocking answers only the first.
Setting up a redocking test
Choose a complex whose ligand is well resolved, strip the ligand from the receptor and prepare the protein as you will for the real project: the same protonation, the same decision about waters and cofactors, the same box size. A redocking test run on a cleaned-up receptor says little about a screen run on a raw one.
Build the ligand from a SMILES string or a fresh 3D conformer, not from the crystal coordinates. Starting from the crystal conformation hands the program part of the answer. Center the box on the crystal ligand, as the Automatic option in AutoDock Vina does, and keep its size realistic: the PoseBusters benchmark gave Vina and Gold a 25 Å box so that they stayed comparable with blind docking methods.[2] Repeat with a few random seeds, since a pose that appears in one run only is not a result.
Vina's own RMSD columns are the usual source of confusion. They are measured against mode 1, so a table of five poses with small values only says that the poses resemble each other. Accuracy needs the crystal ligand and a separate calculation.
Where the 2 Å rule comes from
The 2 Å cutoff is a convention. The AutoDock Vina paper notes that it is often used as the criterion for a correctly predicted bound structure, and the PoseBusters authors call it arbitrary but commonly used and recommended for regular-size ligands.[1][2]
The GNINA pose score is defined around it: CNNscore is the network's probability that a pose lies within 2 Å of the true one.[3] Nothing physical happens at 2.0 Å. The line exists so that studies can report one comparable number, the share of top-ranked poses on the correct side of it.
How the RMSD is measured matters as much as the cutoff. For docking it is computed over the ligand's heavy atoms in the receptor frame, with no fitting of the ligand onto the crystal ligand, and with equivalent atoms in symmetric rings matched in the best way. Each choice moves the imatinib numbers in our RMSD analysis: fitting first turns reversed poses at 12 to 13 Å into ones at 1.8 to 2.4 Å, and ignoring symmetry adds up to 0.4 Å to a correct pose.
Cross-docking adds a step. The receptors must be superimposed first, on the binding site, so that every pose and the reference ligand share one frame; only then is the ligand measured, still without fitting. We superimposed every receptor below on 1IEP using the Cα atoms of the 32 residues within 6 Å of imatinib.
The number also has limits, and the worked example below makes them visible. RMSD is an average, so a pose can pass while part of the ligand sits somewhere else: the nilotinib-structure pose in the image at the top of this page has an RMSD of 1.26 Å, yet atoms of its piperazine ring are up to 3.75 Å from their crystal positions. RMSD also does not scale with ligand size, the weakness that length-normalized scores such as TM-score were designed to fix for proteins, and it says nothing about whether the key hydrogen bonds are there. PLIP lists the contacts of a pose for that check.
For protein-protein docking, DockQ plays the role of the 2 Å rule and the DockQ tool computes it.
Redocking versus cross-docking
In a real project the receptor was usually crystallized with a different ligand, or has no ligand at all. Docking into such a structure is cross-docking, and it is harder because the pocket has already adjusted to the other molecule. In the GNINA 1.0 benchmark, the top Vina pose was within 2 Å in 58% of redocking cases and 27% of cross-docking cases; rescoring with GNINA's neural networks raised those to 73% and 37%.[3] Cross-docking is the number to expect on a new ligand, and redocking is the ceiling.
Table 1 shows the same effect for one ligand. Imatinib was docked into four Abl structures, each prepared the same way, and scored against the imatinib pose in 1IEP.
| Receptor | Ligand in that structure | Site Cα RMSD to 1IEP | Top Vina score | Top pose RMSD | Poses within 2 Å |
|---|---|---|---|---|---|
| 1IEP | Imatinib | 0 Å (reference) | −12.64 kcal/mol | 0.75 Å | 1 of 4 |
| 2HYY | Imatinib, another crystal | 0.19 Å | −12.45 kcal/mol | 0.85 Å | 1 of 5 |
| 3CS9 | Nilotinib | 0.21 Å | −12.25 kcal/mol | 1.26 Å | 1 of 5 |
| 2GQG | Dasatinib | 1.42 Å | −9.32 kcal/mol | 10.03 Å | 0 of 5 |
The first three structures give the same answer. Their binding sites agree with 1IEP to within 0.21 Å and the top pose is correct, though it drifts from 0.75 to 1.26 Å.
The fourth is the dasatinib complex, which its authors describe as the activated kinase domain, while imatinib is known to bind an inactive conformation.[14][11] Its site is 1.42 Å from 1IEP, the top pose is 10 Å from the crystal pose and the best score is 3 kcal/mol weaker than in the other three. We did not test why the pocket rejects imatinib. The score gap was the visible warning; nothing in the geometry of the wrong poses gave them away.
The run is one ligand on one target. It shows how the checks behave, not how often a setup succeeds. The public ProteinIQ example in Figure 1 puts its top pose 0.94 Å from the crystal pose for the same ligand and pocket, with a different receptor preparation. Both results are correct; the gap between them is the preparation noise that a 2 Å cutoff has to absorb.
What redocking cannot show
Redocking tests the search and the pose ranking in the best case. The receptor was crystallized with this ligand, so the pocket already fits it, and the answer is known to the person running the test.
Warren and colleagues evaluated 10 programs and 37 scoring functions on eight proteins. Every program produced a ligand conformation similar to the crystal pose for at least one target, but the scoring functions were worse at picking that conformation out of the docked poses, and none made a useful prediction of binding affinity.[7]
That separation shows up in the CASF-2016 benchmark: Vina chose a pose within 2 Å for 90.2% of complexes when the native pose was among the candidates, while its correlation with measured affinity was R = 0.604.[5] A setup that redocks well is not a setup that ranks compounds well, because a docking score is not a binding affinity. The three c-Abl inhibitors in the public Vina example show it: Vina ranks them in reverse of their measured affinities.
Benchmarks of learned docking methods add a second trap. Deep learning models trained on PDBbind can look excellent on test complexes similar to their training data. On the PoseBusters Benchmark set, built from complexes released from 2021, the authors report that for targets with low sequence identity to the training data, the deep learning methods produced almost no physically valid poses within 2 Å.[2] Check what a method was trained on before trusting a redocking number from its paper.
Enrichment validation with actives and decoys
When the goal is virtual screening, the question changes from "is this pose right" to "does the setup rank binders above everything else". The standard test docks known actives together with decoys, molecules that match the actives in size, charge and polarity but are not expected to bind, and checks whether the actives rise to the top. DUD-E, the common benchmark, has 22,886 actives against 102 targets and 50 property-matched but topologically dissimilar decoys per active.[8]
Two numbers summarize a run. The area under the ROC curve (AUC) is the probability that a random active is ranked above a random decoy, with 0.5 meaning no information. The enrichment factor at 1% (EF1%) compares the hit rate in the top 1% of the ranked list with the hit rate of the whole library. If 20 of the top 100 compounds in a 10,000-compound library with 100 actives are active, the hit rate is 20% against a library average of 1%, so EF1% is 20.
AUC averages over the whole list, but screening only uses the top of it, which is why Truchon and Bayly showed it to be a poor measure of early recognition and proposed BEDROC instead. They also showed that EF does not tell apart rankings on either side of its cutoff.[9] Report both, and read EF against its ceiling: with 50 decoys per active, a library is 1.96% active, so EF1% cannot exceed 51.
Scores here are modest. On DUD-E, Vina 1.2 averaged an AUC of 0.72 and put the crystal pose first for 68% of complexes.[4] DUD-E also carries analogue and decoy bias, and neural networks trained on it can score well without learning interactions. Models trained on PDBbind did no better than Vina on it in the same analysis.[10] Treat an excellent AUC from a learned scorer on DUD-E with suspicion, and test on the target and chemical series that you will screen.
PoseBusters: what the checks are
A pose can sit exactly where the crystal ligand does and still be impossible, because RMSD says nothing about bond lengths or clashes. PoseBusters, a Python package built on RDKit, tests that the ligand is a valid molecule, that its geometry is plausible and, if a receptor is supplied, that it does not collide with the protein.[2] A pose that passes every test is called PB-valid. The PoseBusters tool on ProteinIQ runs version 0.6.5 and returns one row per pose.
The paper lists 18 tests in three groups. The current package splits some of them by cofactor type, so in Redock mode it reports 28 pass or fail columns, including the three file-loading checks and the 2 Å RMSD test. Table 2 gives the checks and the thresholds from the paper and the 0.6.5 configuration.
| Group | Checks | Passes when |
|---|---|---|
| Chemistry | Loading, sanitization, InChI conversion, connectivity, no radicals | RDKit can read the molecule and it is one connected, valid structure |
| Identity | Molecular formula, bonds, double-bond stereochemistry, tetrahedral chirality | The pose is the same molecule as the reference ligand, including its stereoisomer |
| Bond geometry | Bond lengths, bond angles | Within 0.75 to 1.25 times RDKit's distance-geometry bounds |
| Ligand clashes | Internal steric clash | No non-bonded atom pair closer than 0.7 of its lower bound |
| Flatness | Aromatic rings, double bonds, non-aromatic six-membered rings | Aromatic rings and double bonds within 0.25 Å of a plane; saturated six-membered rings not planar |
| Strain | Internal energy | UFF energy at most 100 times the mean of 50 generated conformers |
| Protein contact | Maximum distance, minimum distance, volume overlap; also cofactors and waters | The ligand lies within 5 Å of the protein, no atom pair is closer than 0.75 times the sum of its van der Waals radii, and volume overlap is under 7.5% |
| Reference | RMSD within 2 Å | Symmetry-aware heavy-atom RMSD to the crystal ligand of 2 Å or less |
The PB-valid rate is the share of poses that pass everything. It is reported alongside the RMSD success rate, and for some methods the two differ greatly. On the Astex Diverse set, DiffDock put 72% of top poses within 2 Å but only 47% were also PB-valid; Uni-Mol went from 45% to 12% and TankBind from 59% to 5.9%. Gold and Vina barely moved, from 67% to 64% and from 58% to 56%.[2]
Within 2 Å but invalid, or valid but wrong
The two failures run in opposite directions, and we can show both. First, a pose can be valid and wrong. In the imatinib runs above, every one of the 19 Vina poses passed all 27 non-RMSD checks, including the 16 that were 6 to 15 Å from the crystal pose. Reversed end to end or turned in the pocket, they are still reasonable molecules in a reasonable place. A clean PoseBusters report says nothing about whether the pose is the right one.
Second, a pose can be accurate and invalid. Vina's poses did not show this, which fits the Astex result above, so we constructed it: the crystal imatinib with random coordinate noise added to every atom. This is a control, not a docking result, and it mimics the bond distortions that some learned methods produce.
| σ per coordinate | RMSD to crystal pose | Checks failed |
|---|---|---|
| 0.1 Å | 0.15 Å | None |
| 0.2 Å | 0.32 Å | Bond lengths, bond angles |
| 0.3 Å | 0.50 Å | Bond lengths, bond angles, internal steric clash, aromatic ring flatness, minimum distance to protein |
| 0.4 Å | 0.67 Å | Bond lengths, bond angles, aromatic ring flatness, internal energy |
At 0.32 Å from the crystal pose, one-sixth of the 2 Å limit, six bonds and an angle were already out of range. RMSD counts such a pose as a success, and PoseBusters does not. What PoseBusters cannot do is judge the science: it does not score binding, model induced fit or know whether the right waters are present, so a PB-valid pose is a necessary condition and never a sufficient one.
A validation workflow
Run the checks in an order that spends effort only after the cheaper ones pass.
- Redock the co-crystallized ligand of your target structure with the settings you plan to use, and measure a symmetry-corrected RMSD in the receptor frame. If the top pose is not within 2 Å, fix the preparation, box or protonation before anything else.
- Run PoseBusters in Redock mode on all poses. A pose that is within 2 Å but fails a test is not a success.
- If you will dock into a single structure, cross-dock a few ligands from other crystal structures of the same target into it, after superimposing the receptors on the site. Choose the receptor, or the set of receptors for ensemble docking, that keeps the success rate highest.
- For screening, dock known actives and property-matched decoys for your target, and read AUC and EF1% together. A decoy set from the same chemical series as your library is worth more than a generic one.
- Set a score cutoff from those results, not from a published number, then dock the real library. Run PoseBusters on the top hits and check their contacts with PLIP before buying or synthesizing anything.
Reading the results together
Table 4 turns the combinations of results into the next step.
| What you see | What it usually means | What to do |
|---|---|---|
| Redocking fails (top pose beyond 2 Å) | The preparation or box is wrong, or the ligand needs a tautomer, protonation state, water or cofactor that was left out | Fix the receptor and ligand preparation; try more seeds and a higher exhaustiveness before changing the program |
| Redocking passes, cross-docking fails | The receptor conformation suits only its own ligand | Use a structure closer to your ligands, or dock into several receptors and take the best |
| Within 2 Å but fails PoseBusters | The pose is close but distorted or clashing | Do not count it; minimize it with the protein fixed, as the PoseBusters authors did, then test again |
| Passes PoseBusters but is beyond 2 Å | A plausible but wrong binding mode | Look at lower-ranked poses, rescore with a second method, check contacts |
| Pose success is good, AUC near 0.5 | The setup finds poses but its scores do not separate binders from decoys | Rescore with another function or a learned scorer; do not use the scores for ranking |
| High AUC, poor poses | Scores may be tracking ligand size or other dataset bias, not the interactions | Check poses and test on a decoy set built for your target |
| Everything passes on one target | The setup works for that target and ligand type | Validate again for each new target, receptor structure and chemical series |
Validation tools on ProteinIQ
Every step of the workflow above runs on ProteinIQ (Table 5). One caution: the RMSD Calculator pairs atoms by name and does not correct for symmetry, so use the PoseBusters RMSD test for ligands with symmetric rings.
| Task | Tool |
|---|---|
| Redock and cross-dock with a box built from a crystal ligand | AutoDock Vina, smina |
| Rescore poses with a neural network | GNINA |
| Generate poses without a defined box | DiffDock |
| Check chemistry, geometry, clashes and the 2 Å criterion | PoseBusters |
| Compare a pose with a crystal ligand by atom names | RMSD Calculator |
| List hydrogen bonds and other contacts in a docked complex | PLIP |
| Predict a protein-ligand complex and affinity | Boltz-2 |
| Score a predicted protein-protein complex against a reference | DockQ |
Frequently asked questions
What is redocking in molecular docking?
Redocking removes the ligand from a crystal structure of a complex, docks it back into the same protein and compares the top pose with the crystal pose. A top pose within 2 Å RMSD counts as a success. It tests whether the docking setup can reproduce a pose that is known to be right.[1]
What RMSD counts as a successful docking pose?
The convention is 2 Å or less, measured over the ligand's heavy atoms in the receptor frame, without fitting the ligand onto the crystal ligand and with symmetry corrected. The authors of PoseBusters describe the cutoff as arbitrary but commonly used and recommended for regular-size ligands.[2]
What is the difference between redocking and cross-docking?
Redocking uses the receptor structure that the ligand was crystallized with. Cross-docking uses a structure crystallized with a different ligand, and in the GNINA benchmark the share of correct top poses fell from 58% to 27% for Vina scoring on that change.[3]
What does PB-valid mean?
A pose is PB-valid when it passes every PoseBusters test, covering chemistry, bond geometry, internal clashes, ring and double-bond flatness, strain and contact with the protein. It says nothing about whether the pose is in the right place, so it is reported next to the share of poses within 2 Å.[2]
Can a pose within 2 Å still be wrong?
Yes, in two ways. It can contain distorted bonds or clashes and fail PoseBusters, as when DiffDock's 72% within 2 Å on the Astex set fell to 47% once validity was required.[2] And an average under 2 Å can hide one end of the ligand sitting several ångströms from its crystal position, as in the cross-docked nilotinib-structure pose above.
Does a passed redocking test mean the docking scores are reliable?
No. It tests pose reproduction in the best case. Scoring functions that pick the right pose still correlate only moderately with measured affinity, and a docking program can pass redocking and fail to separate actives from decoys.[5]


