v1.1.2 randomly changes its output without any valid cause

[YuichiMotai]

Bug description

Uni-Dock ver 1.1.2 has unstable behaviour, randomly changing its output without any valid cause. We made the following reproduction case. The behavior is similar to that with a missing synchronization in CUDA kernel.

Small reproduction case

We added a small non-affecting if-statement that never enters in one of the kernel functions, and observe an abnormal behaviour.

Preparation

We first modified the source code to produce exactly identical output with respect to identical data by:

1. Fixing the initial value f0 = 3.0 in bfgs_warp() in wap_ops.cuh (3.0 would be arbitrary)
2. Commenting out #pragma omp parallel for and #pragma omp critical in main() in main.cpp
3. Adding line sort(ligand_names.begin(), ligand_names.end()); in main() between:
   • std::cout << "Total ligands: " << ligand_names.size() << std::endl;
   • if (score_only) {
4. Fixing batch size in template_batch_docking() by adding
   • && batch_size < MAX_BATCH_SIZE) { at the end of while-loop between
     • while (predict_peak_memory<Config>(batch_size, exhaustiveness, all_atom2_numbers,
     • batch_ligands.emplace_back(
   • with MAX_BATCH_SIZE = 150

After these changes and with fixed seed, we confirmed that unidock generates identical results every time it's called on a fixed, sorted data.

BTW, this could be a solution of this issue (https://github.com/dptech-corp/Uni-Dock/issues/1330).

Small change

However, when we added a never-entering if statement in set() in common.cuh, after second call to local_to_lab(), calculating and writing updated atom coordinates to m_coords, we observe the generated results are no longer matching.

The added if statement is never supposed to be entered nor touch any data/variable, hence we expect no change in the output. There is no sign of the if condition entered as we can find no

====================== NOT SUPPOSED TO ENTER HERE ======================

in the output file thats is otherwise printed.

Environment and data

• Uni-Dock: ver 1.1.2 from source
• Data: DUD-E ligands with target ABL1
• CPU: 12th Gen Intel(R) Core(TM) i9-12900
• GPU: NVIDIA GeForce RTX 4060 Ti
• CUDA: 12.2
• OS: ubuntu 22.04

Additional Info.

If you have trouble reproducing events, would you mention me here. We could share data and programs with you.

Thank you.

[matija-marijan]

Hi @YuichiMotai, appreciate the work that you've done, and thanks for mentioning my issue. I'm curious to what "fixed, sorted data" means in:

After these changes and with fixed seed, we confirmed that unidock generates identical results every time it's called on a fixed, sorted data.

If you could elaborate a bit on that, it would be helpful. Thanks.

Also, after building Uni-Dock with your proposed fixes, I am still not able to get reproducible results. If i call Uni-Dock for the same 5 ligands within one --ligand_index folder, i get different results, but after i call it another time, i get the identical five (different) results. Have you found this in your experiments as well perhaps?

Could you share the data and programs you used for fixing the issue?

[YuichiMotai]

Hi @matija-marijan, sorry for the delay
Here is our code modified from ver 1.1.2.

I'm curious to what "fixed, sorted data" means

Before we make changes to src, unidock gave slightly different standard output in the terminal.
Since we wanted a deterministic result (i.e. we wished to see identical standard ouput in the terminal and output .pdbqt files every time unidock runs on identical data), we applied some changes to the src to have the desired effect.
We encountered the bug when we were playing with common.cuh.

Feel free to let us know if you have any difficulty with our code.

The following code is shared to @matija-marijan via a temporal GitHub repository. The repo. will be deleted after communication.

python scripts

• converter.py
• input_generator.py
• run_dock.py

Modified unidock/src

• common.cuh
• warp_ops.cuh
• main.cpp

Data

• abl1.tar.gz

Directory Structure

• build
• dud-e
  • all
    • abl1
      • contents of abl1.tar.gz
• script
  • converter.py
  • input_generator.py
  • run_dock.py
• unidock
• unidock-tools
• .gitignore
• README.md

How to run our code

1. Place the files in the same way as the above directory structure.
2. Replace common.cuh, warp_ops.cuh, main.cpp with ours
3. build unidock with following "Modified unidock/src"
   • cmake -S unidock -B build
   • cmake --build build -j4
4. run python3 script/input_generator.py
5. run python3 script/run_dock.py

[matija-marijan]

Hi @YuichiMotai, thank you very much for reaching out to me and sending me your code. I have downloaded the scripts you have sent me, and I will test them out for myself to try and recreate the results you have. I will reply in this thread if I come across anything relevant or new.

[matija-marijan]

Hi again @YuichiMotai. Unfortunately, when I build your Uni-Dock fix, it doesn't make a difference when comparing it to my fix, guided by your instructions, from a couple of days ago.
If I call unidock for the same folder containing 5 identical sdf ligands (with different file names), with the same seed, I get five different results [-5.9585 -6.388 -6.252 -6.4615 -6.282500000000001], but the expected output should be [-6.25, -6.25, -6.25, -6.25, -6.25], for example.
Unfortunately, it seems as though this problem lies deeper within Uni-Dock's source code, and it would be helpful if someone could take a closer look at this.
Thanks again for your help!