Error "Not enough RF voltage" in Beam threading

[554sj4606]

Hello,

I hope this email finds you well. I am Satya, a PhD student at CERN working on FCC-ee optics tuning studies with pyAT. I am writing to seek your advice regarding an issue I am encountering with beam threading. Specifically, I applied larger alignment errors of 100 µm on arc magnets and 70 µm on IR magnets, then attempted to run the optics correction algorithm starting with beam threading. However, the algorithm fails at the beam threading step, reporting that the "RF voltage is not enough" (please refer to the attached error). Despite setting the RF voltage correctly, the error persists. Below, I have detailed how the RF voltage was configured. I would greatly appreciate any guidance or suggestions to resolve this issue.

ring=at.load_lattice('FCCee_v23.mat',mat_key='ring')
at.Lattice(ring,energy = 45.6e9,particle='electron')
ring.enable_6d()
ring.harmonic_number = 121200
ring.set_cavity(Voltage=79e6)
ring.set_rf_frequency()
ring.set_cavity_phase()
ring.tapering()

Error:
iteration # 8/100 with 250/5628 BPMs and 251/1876 steerers
Hor. traj. | std: 287 microm | (min, max) (-1.68e+03, 2e+03)
Ver. traj. | std: 203 microm | (min, max) (-1.1e+03, 1.32e+03)
Hor. cor. | std: 0.918 microrad | (min, max) (-5.81, 5.84)
Ver. cor. | std: 0.995 microrad | (min, max) (-9.96, 6.56)
orbit is found? False

iteration # 9/100 with 254/5628 BPMs and 255/1876 steerers
Hor. traj. | std: 113 microm | (min, max) (-1.71e+03, 13.8)
Ver. traj. | std: 45.3 microm | (min, max) (-717, 88.3)
Hor. cor. | std: 0.92 microrad | (min, max) (-5.81, 5.83)
Ver. cor. | std: 0.998 microrad | (min, max) (-9.96, 6.55)
orbit is found? False

iteration # 10/100 with 325/5628 BPMs and 326/1876 steerers
Hor. traj. | std: 291 microm | (min, max) (-1.55e+03, 1.72e+03)
Ver. traj. | std: 367 microm | (min, max) (-1.9e+03, 1.96e+03)
Hor. cor. | std: 1.02 microrad | (min, max) (-5.81, 5.83)
Ver. cor. | std: 1.1 microrad | (min, max) (-9.96, 6.55)
orbit is found? False

iteration # 11/100 with 341/5628 BPMs and 342/1876 steerers
Hor. traj. | std: 101 microm | (min, max) (-866, 881)
Ver. traj. | std: 143 microm | (min, max) (-1.51e+03, 1.64e+03)
Hor. cor. | std: 1.05 microrad | (min, max) (-5.81, 5.83)
Ver. cor. | std: 1.12 microrad | (min, max) (-9.96, 6.55)
Not enough RF voltage: unstable ring
correction trajectory failed
lattice is unstable after correction step: trajectory

[swhite2401]

Hello,
Have you tried doubling the voltage ?

[554sj4606]

Dear @swhite2401,

Many thanks for the response. I did try with 200MV as well and I still get the same message claiming that “not enough RF voltage”

[swhite2401]

Are your using the code from @simoneliuzzo ? Could you send us your lattice file and script that produce this error?

[simoneliuzzo]

Dear @554sj4606,

looking at your output you are using my code commissioning simulations.

Could you share the mat file and the original MADX or MAD8 file?

thank you
best regards
Simone

[simoneliuzzo]

Dear @554sj4606,

in any case looking at your code, the trajectory correction does not converge. Only 341/5628 BPMs see beam.
Could you try to reduce the errors?

import at
import at.plot

ring=at.load_lattice('FCCee_v23.mat',mat_key='ring')
at.Lattice(ring, energy = 45.6e9)
ring.enable_6d()
ring.harmonic_number = 121200
ring.set_cavity(Voltage=79e6)
ring.set_rf_frequency()
ring.set_cavity_phase()
ring.tapering()

ring.plot_beta()

works? (I removed also the "particle = electron", as it is a future feature.

[554sj4606]

Dear @simoneliuzzo,

I even tried applying the small errors only to the arc magnets without any IR magnet errors, and I still see the same error...

[lfarv]

ring=at.load_lattice('FCCee_v23.mat',mat_key='ring')
at.Lattice(ring, energy = 45.6e9)

Nothing to do with the problem itself, but the 2nd line is completely useless: you create a copy of ring, but you don't assign it to anything, so it's lost ! Remove this line !

[lfarv]

Now I understand: if you want to change the energy, either use:

ring=at.load_lattice('FCCee_v23.mat', mat_key='ring', energy=45.6e9)

or

ring=at.load_lattice('FCCee_v23.mat',mat_key='ring')
ring.energy = 45.6e9

Maybe a wrong energy is a reason to miss RF voltage.

[554sj4606]

files.zip
Dear all,

Many thanks for your inputs. I have tried using setting the energy inside the load_lattice function, but I sill see the error.
Here I am attaching the mat and madx lattice files as well as the script used to run the correction algorithm. Please let me know if you find any error in my script.

[lfarv]

Hi @554sj4606
I loaded your lattice.
The energy in the base lattice is already 45.6 GeV. No need to reset it, but what you are doing is OK.
The energy loss is 39.089584 MeV. So 200 MV of RF is far enough. The original value of 79.2 MV is already sufficient.
After turning 6d ON, tuning the cavities, the lattice is perfectly correct, ring.plot_beta() succeeds.
So the lattice that you are saving back in "lattice.mat" is correct. What goes wrong happens later in the commissioning simulation code.
@simoneliuzzo, it's up to you !

[simoneliuzzo]

Dear @554sj4606,

in the commissioning simulatios code, if the input lattice is ok (see comments from @lfarv) then the issue is either with errors or correctors. From the output, the lattice is unstable, the beam threading is not succeeding.
you could try:

• smaller errors (try 1e-8, just as an extreme case)
• verify the correctors and BPMs distribution
• verify the correct response matrices are use and that those have no "Nan"
• try to change the parameters of the trajectory correction
  Since the AT part of this issue is solved, we can move the discussion to an issue in: https://gitlab.esrf.fr/BeamDynamics/commissioningsimulations/-/issues

[simoneliuzzo]

Dear @554sj4606,

I tried your script, reducing the errors to 1e-6 in all elements. All works ok.

In order to compute the orbit response I had to reduce the kick, or some orbits were not computed.
I will now try with bigger errors to see the trajectory correction take place.

[simoneliuzzo]

Dear @lfarv,

is is possible that find_orbit() for unstable rings, now trows an error instead of returning nan orbits?

This is apparently what causes the problem in the commissioning simulations code of @554sj4606 .

I will try reverting AT to an older release to understand when the change occurred.

[lfarv]

find_orbit does not even care about stability, it just relies on tracking over one turn. So the error should come from tracking, which may happen for many reasons. What error do you get ?

[simoneliuzzo]

import at
import numpy as np
import at.plot
import matplotlib.pyplot as plt


ring=at.load_lattice('FCC_V23/FCCee_v23.mat',mat_key='ring')


ring.enable_6d()
ring.harmonic_number = 121200
ring.set_cavity(Voltage=200e6)
ring.set_rf_frequency()
ring.set_cavity_phase()
ring.tapering()

n_elements = len(ring)

ind_bpm = at.get_uint32_index(ring, at.Monitor)

ind = at.get_uint32_index(ring, at.Quadrupole)
Nel = int(len(ind))

dx = np.zeros(n_elements)
dy = np.zeros(n_elements)


for err in np.arange(0.0, 1e-4, 1e-5):

    dx[ind] = dx[ind] + err * np.random.randn(Nel)
    dy[ind] = dy[ind] + err * np.random.randn(Nel)

    # apply errors
    at.set_shift(ring, dx, dy)

    new_guess, orb = at.find_orbit(ring, ind_bpm) # , guess=np.zeros((1, 6)))
    print(orb)

here the small script to reproduce the error:

Traceback (most recent call last):
  File "/machfs/liuzzo/FCC/V23_z/minimial_example.py", line 36, in <module>
    new_guess, orb = at.find_orbit(ring, ind_bpm) # , guess=np.zeros((1, 6)))
  File "/machfs/liuzzo/FCC/V23_z/fcc23_venv/lib/python3.9/site-packages/at/physics/orbit.py", line 511, in find_orbit
    return find_orbit6(ring, refpts=refpts, **kwargs)
  File "/machfs/liuzzo/FCC/V23_z/fcc23_venv/lib/python3.9/site-packages/at/lattice/cavity_access.py", line 48, in wrapper
    return func(ring, *args, **kwargs)
  File "/machfs/liuzzo/FCC/V23_z/fcc23_venv/lib/python3.9/site-packages/at/physics/orbit.py", line 449, in find_orbit6
    orbit = _orbit6(ring, keep_lattice=keep_lattice, **kwargs)
  File "/machfs/liuzzo/FCC/V23_z/fcc23_venv/lib/python3.9/site-packages/at/physics/orbit.py", line 324, in _orbit6
    _, dt = get_timelag_fromU0(ring, method=method, cavpts=cavpts)
  File "/machfs/liuzzo/FCC/V23_z/fcc23_venv/lib/python3.9/site-packages/at/physics/energy_loss.py", line 179, in get_timelag_fromU0
    raise AtError('Not enough RF voltage: unstable ring')
at.lattice.utils.AtError: Not enough RF voltage: unstable ring

[simoneliuzzo]

so in fact, there are still the NAN, but now also the error, and this is new to me

[simoneliuzzo]

I reverted to pyat-0.5.0 and the error message does not appear. only NAN

[lfarv]

The error comes from get_timelag_fromU0. This is called to get an approximation of the phase of the 6d orbit if there is no guess. I don't understand how you can miss RF voltage: could the radiated power be so different from the bare lattice, or could the voltage of you test lattice be different from the nominal one ?

[lfarv]

I ran you script, on the master branch, and I don't get errors:

[[ 8.12294532e-09  2.34084963e-09  0.00000000e+00  0.00000000e+00
  -1.04958225e-04 -1.94740998e-05]
 [ 1.27519643e-08  4.95262364e-09  0.00000000e+00  0.00000000e+00
  -1.04958225e-04 -1.94740998e-05]
 [ 2.03406781e-08  6.67875135e-09  0.00000000e+00  0.00000000e+00
  -1.04958225e-04 -1.94740998e-05]
 ...
 [ 3.13156056e-09 -4.00125068e-10  0.00000000e+00  0.00000000e+00
  -1.04958225e-04 -1.94740998e-05]
 [ 2.97864494e-09  2.02638340e-10  0.00000000e+00  0.00000000e+00
  -1.04958225e-04 -1.94740998e-05]
 [ 3.35355982e-09  8.34121952e-10  0.00000000e+00  0.00000000e+00
  -1.04958225e-04 -1.94740998e-05]]
-----------
[[nan nan nan nan nan nan]
 [nan nan nan nan nan nan]
 [nan nan nan nan nan nan]
 ...
 [nan nan nan nan nan nan]
 [nan nan nan nan nan nan]
 [nan nan nan nan nan nan]]

And so on, 10 times.
Which version are you running ?

[simoneliuzzo]

For this test I did a new environment this morning and

pip install accelerator-toolbox

[lfarv]

there are still the NAN, but now also the error,

Can you explain ? Either you get an error, or you get a result, it cannot be both…

pip install accelerator-toolbox

So release 0.6.1. I'll try this one !

[simoneliuzzo]

I restored the main release. error is not reproduced anymore.

I recreated the environment as this morning from zero (pip install --upgrade pip, pip install accelerator-toolbox, pip install matplotlib). Error is not reproduced anymore.

So, I am unable to reproduce the error at the moment.

in the bad case: find orbit would return NaN AND the RF voltage error , reported by Satya.

But for now, I may not reproduce it. So it is fixed by pure conversation.

[simoneliuzzo]

Here the list of actions that makes the script of @554sj4606 work:

python -m venv fcc23_venv_2
source fcc23_venv_2/bin/activate
pip install --upgrade pip
pip install accelerator-toolbox
pip install matplotlib
cd commissioningsimulations/
pip install -e .
cd ..
/machfs/liuzzo/FCC/V23_z/fcc23_venv_2/bin/python /machfs/liuzzo/FCC/V23_z/FCC_V23/script_v23.py

for orbit correction to work, dcorL in /commissioningsimulations/correction/ClosedOrbit.py has to be changed to 1e-7 (line 459) I will push this change to commissioning simulations (it is unrelated to this issue).

[simoneliuzzo]

Dear @554sj4606,

I take it back. The error is still there when running the script

[lfarv]

So it is fixed by pure conversation.

We should talk more !
Seriously, that's worrying, because both Satya and you got he same problem, so it's pending somewhere.
I did the same as you, pip install accelerator-toolbox. Except that I did not install commissioningsimulations. Can it be the problem ?

NaN AND the RF voltage error

Impossible. If there is an exception, find_orbit does not return at all.

[simoneliuzzo]

Dear @lfarv,

I should have copied the full output but I thought the minimal_script.py would make the error reproducible.

You are correct, in fact the output was:

very small numbers for zero errors
NAN for 1e-4 errors
RF voltage error for 2e-4 errors

[simoneliuzzo]

Dear @lfarv and @554sj4606,

I think the problem is in find_orbit(). However I am unable to reproduce the error outside commissioning simulations. This morning I could. Now not anymore.

So I will modify commissioning simulations code to protect from this issue, in order to let Satya continue his work.

[simoneliuzzo]

Here the running script
script_v23_modSML.py.zip

and the full reinstallation/update sequence:

git clone https://gitlab.esrf.fr/BeamDynamics/commissioningsimulations.git
deactivate
conda deactivate
python -m venv fcc23_venv_2
source fcc23_venv_2/bin/activate
pip install --upgrade pip
pip install accelerator-toolbox
pip install matplotlib
cd commissioningsimulations/
pip install -e .
cd ..
./fcc23_venv_2/bin/python script_v23_modSML.py

please @554sj4606 let me know if it works

[554sj4606]

Dear @simoneliuzzo and @lfarv,

I ran the script (with small alignment errors) with the workflow suggested by simone. Although I see the same error message as before, the beam threading continues to run (please see the attachment below).

But when I tried to apply really large errors of 100um, the beam threading fails claiming the lattice is unstable.
I even tried to apply the errors in an adiabatic way. For example: turn off sextupoles, apply arc quads and sextupole errors and run traj. and orbit correction, then apply dipole errors run traj. and orbit correction and finally IR errors and so on. Still, I cannot manage to run the correction with large errors. I encounter this problem when I use v23 lattice, and the v22 lattice works fine so far for the errors applied in adiabatic manner.

@simoneliuzzo May I know if you managed to apply large errors?

iteration # 1/50 with 24/5628 BPMs and 25/1876 steerers
Hor. traj. | std: 241 microm | (min, max) (-654, 654)
Ver. traj. | std: 495 microm | (min, max) (-1.21e+03, 1.13e+03)
Hor. cor. | std: 0.126 microrad | (min, max) (-4.08, 0.794)
Ver. cor. | std: 0.152 microrad | (min, max) (-5.94, 1.12)
Not enough RF voltage: unstable ring
orbit is found? False

iteration # 2/50 with 254/5628 BPMs and 255/1876 steerers
Hor. traj. | std: 320 microm | (min, max) (-1.57e+03, 1.1e+03)
Ver. traj. | std: 414 microm | (min, max) (-879, 1.95e+03)
Hor. cor. | std: 0.176 microrad | (min, max) (-4.08, 1.45)
Ver. cor. | std: 0.209 microrad | (min, max) (-5.94, 1.79)
Not enough RF voltage: unstable ring
orbit is found? False

iteration # 3/50 with 449/5628 BPMs and 450/1876 steerers
Hor. traj. | std: 158 microm | (min, max) (-1.08e+03, 579)
Ver. traj. | std: 145 microm | (min, max) (-750, 763)
Hor. cor. | std: 0.199 microrad | (min, max) (-4.08, 1.45)
Ver. cor. | std: 0.229 microrad | (min, max) (-5.94, 1.79)
orbit is found? False

iteration # 4/50 with 493/5628 BPMs and 494/1876 steerers
Hor. traj. | std: 74.1 microm | (min, max) (-920, 717)
Ver. traj. | std: 81.4 microm | (min, max) (-834, 829)
Hor. cor. | std: 0.215 microrad | (min, max) (-4.08, 1.45)
Ver. cor. | std: 0.237 microrad | (min, max) (-5.94, 1.79)
orbit is found? True

[simoneliuzzo]

Dear @554sj4606 ,

no I did not try. It could be that V23 lattice is simply more sensitive to errors.

Take the same script, just trajectory, orbit and tune correction. Change only the input file. at which error amplitude the correction fails for V22 and V23? for which orbit amplitude?

We can continue the discussion via email, since the issue is not reproducible within AT alone.

[554sj4606]

Dear all,

I appreciate all your efforts. I will try what @simoneliuzzo suggested and let you know if it works

[swhite2401]

hello all, I have also tried running @simoneliuzzo script on my side, I have python 3.11 and numpy 1.24.2 and the pyAT master branch.
I have a different error:

    b_over_a = numpy.linalg.solve(a, b)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "<__array_function__ internals>", line 200, in solve
  File "/Users/simonwhite/miniforge3/envs/pyat_env/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 386, in solve
    r = gufunc(a, b, signature=signature, extobj=extobj)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simonwhite/miniforge3/envs/pyat_env/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 89, in _raise_linalgerror_singular
    raise LinAlgError("Singular matrix")
numpy.linalg.LinAlgError: Singular matrix

I have added the following lines in orbit.py:

        try:
            b_over_a = numpy.linalg.solve(a, b)
        except numpy.linalg.LinAlgError:
            b_over_a = numpy.zeros(6)
            b_over_a[:] = numpy.nan

Which seem to solve this issue. However, I could not produce any error relating to RF voltage.
@lfarv: the reason is that a can have nans even when tracking for only a turn, should we add this additional protection in any case?

[lfarv]

@swhite2401 : There can indeed be several reasons to get nan in tracking over one turn, for instance aperture limitations. In such a case, the error thrown is fully justified. There cannot be any closed orbit, and I don't think that catching the error and returning nan arrays can help in any way: there is nothing you can do without a closed orbit.

For me, the best improvement would be to rewrite the error message to make it more understandable than "singular matrix". Something like: "Particle lost: Impossible to find a closed orbit"?

[lfarv]

Following @swhite2401's test, I have a suspicion: the failing tracking over 1 turn could break the computation of the energy loss itself. If the energy loss comes out as NaN (to be checked), the test Voltage >= losses fails, producing the error. if this is the problem, what should we do ?

I think that get_energy_loss should indeed throw an error, but with a mode explicit message: "Particle lost: impossible to compute the energy loss" for instance.

Then in find_orbit:

• either we propagate the error and everything stops,
• or we catch the error, set the "guessed" orbit to zeros(6) and try the closed orbit computation. It will likely end up with the error that @swhite2401 experienced: "Particle lost: Impossible to find a closed orbit", but maybe not…

In any case, we need to improve the error message so that the user understands what happens ("missing RF voltage" is misleading). But the error itself is justified: there is nothing you can do if tracking over 1 turn fails, and for the caller, catching the error is as easy as checking for nans.

[swhite2401]

@lfarv , the is a protection on the energy loss calculation, if tracking one turn fails then it is performed element by element so in principle this cannot fail... but it is true that it is the only thing I can think of that would produce this error (which I could not reproduce by the way...) .... more tests needed.

Concerning the failed closed orbit calculation I have no strong opinion on that but for sure the error message needs to be improved.

[swhite2401]

@lfarv, throwing an exception instead of returning NaNs would break backward compatibility and require to modify all the code that depend on pyAT, since closed orbit is rather central we should be very very careful if modifying its behavior.

What I would propose is to fix the bug maintaining backward compatiblity now and then propose the "cleaner" solution for the next major release. Could we use this __future__ import for this as well?
I don't think we have a clear strategy to make these more severe, programmed changes....

[lfarv]

@swhite2401 : I fully agree with your comment, but what I proposed did not change anything: presently we have the error "missing RF voltage". I just suggest to replace that with a better message, if needed. And optionally to catch the error and try to go on anyway with orbit computation.

However, considering your last remark on element by element tracking computation, the possibility of having NaN in the energy loss looks very unlikely, but not completely excluded (like r_in[0] correct, but r_in[4] == NaN). On the other hand, If the reported error is right, the reasons to get what @554sj4606 and @simoneliuzzo reported is that:

• either the errors change drastically the radiation losses (more that a factor 5),
• or something breaks the RF voltage

[swhite2401]

So there are 2 case:
-SVD error -> we may return NaNs as was done in the past
-missing RF voltage: there only case I can think of that would produce this error is a wrong calculation of U0, in this case We could just catch in the the function that computes energy loss. An additional improvement if the tracking method fails would be to fall back to the analytical one with a warning.