K-vector Search Discussion Notes

[Kvieta1990]

04102024

---

Agenda

---

• Documentation needed for the k-vector search functionality -> howto?

• How are those ARM32 binaries built? Do we have dedicated machines or virtual machines with the specific architecture?

• I created a branch called develop from the main master branch and added in our new implementation of the k-vector search functionality. I tested it out, fixed several little bugs and I am now confident that the functionality is working robustly. This also includes the change of the k-vector search routine itself, which does need a bit of attention as detailed below.

• Through testing against artificially generated data (for which we know exactly the ground truth k-vector), we found that we have to go with a fine k-grid for the k-vector searching. This proposes the need for fast calculation. Concerning the speedup, we introduced a combination of two types of solutions -- using multiprocessing together with Cython.

• Question for @briantoby -- are we currently using any Cython in GSAS-II? Basically, a Cython file is something pretty much like Python code but can be compiled as C library to be imported into Python routine, which can significantly boost up the calculation. So I implemented multiprocessing in Cython and considering the two layers of speedup, we can now perform a heavy calculation with relatively acceptable amount of time (but still on the level of several hours).

• Testing routine needed for internal people to test out the implementation before releasing to public. I will work on creating a script for people to easily test out the new implementation, with the develop branch.

---

Minutes

---

• Add the Cython compiling stuff into the local build scripts

  Create new files for the new implementation

  Refer to those files that are not starting with G2 here

• GSAS-II Web Documentation (here) contains general introduction to GSAS-II, meaning it is NOT the place to host the technical details.

  Refer to the source codes here for building the documentation with sphinx.

• The repo here is for building up the tutorial website. We need to manually populate HTML files for new tutorials (by creating its own directory or putting into existing ones). If a new directory is created to host the new tutorials, we need to follow the instructions here to worry about getting the new directory into the website structure.

• The GSAS-II development documentation (see here) is built from source RST files together with the docstring in GSAS-II source codes. The source RST files can be found here. To populate the docstring in GSAS-II python source codes into the rendered documentation, we only need to follow the source code structure and refer to certain classes/methods inside the RST file. The place to search for the source Python files for the doc building purpose is defined here.

  Concerning the documentation for the k-vector search, we want to put the reference to the k-vector search docstring into GSASIIutil.rst file which can be found here.

• For creating the tutorials, relevant exercise files should go into the data subdirectory under certain directory for the specific tutorial. There is an index.htm file inside the data subdirectory -- we don't need to worry about it as it will be automatically generated based on the files we have in the data subdirectory.

• Do a local build for sphinx to make sure the documentation is rendered as expected before pushing to the repo.

• Refer to the GSASIImapvars.py and GSASIIscriptable.rst files for nitty-gritties of using reStructuredText.

04032024

---

Agenda

---

• See the todo list in k_vector_search.py.

• Easy way to have a local test version for beamline scientists to test out, e.g., the k-vector search functionality?

• When clicking on different items in the tree, the menu options will be updated. How?

---

Minutes

---

• Finish up the k-vector search new implementation.

• Create a branch for hosting the new implementation of k-vector and we will work out a way for people to do the testing.

• Brian walked through the implementation of the menu bar items => see notebook for more information.

  https://yr.iris-home.net/gsasiimenunotes

[briantoby]

This is excellent 👍 and we can talk more on all this on Wednesday, but let me respond to as much of this as I can.

I really don't know much about Cython, but I understand it is compiled. The problem will be in how to distribute the platform-specific code. The way things work now is that anything platform-specific for GSAS-II is built in the Scons process and are placed in the binary directory. For distribution for folks who do not build themselves, binaries taken from the .tgz files here: https://github.com/AdvancedPhotonSource/GSAS-II-buildtools/releases/latest. It would be great if we can create and relocate binaries to those locations. If not, we will have to think about how to do this. FWIW, I expect we will "soon" move away Scons to meson. Some work will be needed for Python 3.12 in any case.

If not overly cumbersome, I'd like to keep the all-Python version of your code to be called in try/except block. That way if someone does not have the Cython version installed/configured/... they still have access to the functionality, albeit slow.

Documentation comes in a few flavors. Mostly we show functionality via tutorials. For this you could add a k-vector search to the existing magnetism tutorials or create new one(s). When it comes to explaining how the code works, I tend to put that into either the module or an accompanying .rst file that gets read by SphinxDocs, so either way it ends up in the Code Developer's Manual on ReadTheDocs. We now also have the GSAS-II web pages (here: https://gsasii.github.io/, also generated by Sphinx from .rst and .md files in https://github.com/AdvancedPhotonSource/GSAS-II-tutorials/tree/main/webdocs) but I look at that as for more general stuff like installation notes.

[Kvieta1990]

Thanks a lot Brian for your comments! Really appreciate it! For sure, on Wednesday we can talk more on this.

I have already got those compiled Cython modules in the .pyd format and I do see some .pyd files in the bindist directory and I suppose if I put my compiled .pyd files in the bindist directory, it should work just like whatever other binary files for GSAS-II. Though, I need to do some testing on this and we can discuss. Concerning the try/except process you suggested, for sure I can wrap that around the k-search part. The only thing is that without the Cython modules, the computational time is unacceptably long so I would throw an error message window in case of any exception, telling users to grab those compiled modules. As for how-to, we can discuss and decide.

For the documentation, I will look into the .rst files and will create one for the k-vector search and for sure I can also create a tutorial for it once we make sure everything is working as expected -- as you may already see in my email to people, they can grab the version in the 'develop' branch to test it out and we should get some feedback hopefully soon.

Thank you!

[Kvieta1990]

Just a quick update. I have added in the try/except for the kvec_general module. If the option is 2 (indicating we are going to search over the general k points, which requires heavy computation) and the kvec_general module is not available, we will throw out a message telling users that the module is not available, in which case only option-0 (high symmetry points) and option-1 (high symmetry path) are available.

I already got the compiled binaries for Windows and MacOS with ARM architecture. I will compile on all the other platforms including MacOS with intel chips, Linux OS with intel and ARM chips. We can talk about this on Wednesday about adding them in to the binary release for all the platforms.