How can I use ginkgo reorder/pivoting for cbgmres with ParILUT?

[uboats]

I tried to reorder matrix for better condition for convergence
but didn't find example showing how to use ginkgo reorder/pivoting code with cbgresm and w/ ParILUT
Can anyone help me and show some samples please?

[upsj]

I created an example in #1465 that builds on the preconditioned-solver example. Feel free to ask any questions about it :)

[uboats]

Thanks for the update and details. Looking forward to the enhancement coming soon! :)
I just profiled and found that gko::factorization::ParIlut is very time consuming if fil factor is >= 2 (i.e for matrix with 100k dim, it takes 200s to call gko::factorization::ParIlut).

[upsj]

Is this running on CPU (OpenMP) or GPU (CUDA/HIP)? I think for the use case you describe, it might be most advantageous to compute a ParILUT with a larger number of iterations, and then reuse the sparsity pattern of that factorization for a subsequent ILU factorization with a fixed pattern. The ParILUT algorithm will not be sped up significantly by updating only the values, because it always involves adapting the sparsity pattern as well.

This is currently not possible in Ginkgo, but can be implemented with medium effort.

[uboats]

It's running on reference (CPU).
In my current implementation, the outer iteration is NR iteration, and inside each NR, I follow the example of ilu-preconditioner and use ParILUT preconditioner to replace the original ILU used in example. After applying reorder (thank you again!), I can only need to set ILUT fill-in factor with a much smaller value than before: i.e 1.0~1.4 to converge within 10 GMRES iterations.
And I measured the runtime cost of each stage and find that even with such a small ILUT fill-in factor, the time cost for reorder is tiny, and dominant one is still (100 seconds for 200k dim matrix after ginkgo amd reordering):
par_ilu_fact = gko::factorization::ParIlut<ValueType, IndexType>::build() .with_fill_in_limit(limit) .on(g_exec);
If I understand correctly, the ILU factorization of preconditioner is already finished in above code?
time cost of gmres solver factory is neglectable.
And in next NR iteration, matrix values changed, and I have to wait for another 100 seconds for ILUT. Is it possible to reduce the 100s from 2nd NR iteration?
Thanks again for your patience and nice job!

[upsj]

The Reference implementation is not tailored towards high-performance usage, more for validation. Even running on a single CPU core, OpenMP will be much faster, because we use more efficient algorithms. par_ilu_fact is only a factory, so it does not do anything yet, only once the GMRES factory is being generated on a system matrix will the factorization and resulting preconditioner be generated. With a RCM reordering, the matrix most likely has a band structure that can become dense with pretty few iterations, so this is not surprising. Because the single core ParILUT actually computes the ILU factors exactly (since it is sequential), you probably only need 2-3 iterations.

[uboats]

Got it. I will try omp and I can still use omp_set_num_thread to control the parallelism for omp executor?