Coupling to Default Crate Fan Cooling Program

[msilvafe]

Logging here to not be lost in SLAC. We (UCSD folks) found this in P10R2 and Yuhan + Penn folks found this during the initial LATR highbay cooldown but the fans seems to want to cycle on a ~32 min (this was definitely the cadence in SAT1) and there is a measurable coupling in the superconducting datasets (likely coupled through TES bias DAC voltage reference temperature coefficient) and in the phase monitoring channels (likely coupled through time delays in the smurf having some temperature coupling although that is less understood at this point). Regardless the best solution for operations is to disable the crate fan program and leave the fans running at maximum. We may need to consider sourcing more fan tray spares as this will likely shorten the fan tray lifetime but is a worthwhile investment to remove this source of 1/f.

Some plots included below for reference in case others rediscover this in the future.
From SAT1:

From LATR:

[jlashner]

@swh76 Do you know how we should disable the fan program? If we decide this is the best route forward I can add it to jackhammer.

[msilvafe]

Ahh I should have mentioned that duh...I'm pretty sure these are the commands:

# clia setfanpolicy 20 3 disable
# clia setfanpolicy 20 4 disable
# clia setfanlevel all 15

We should just confirm that the fan trays are always fru 3 and 4 and the max fan speed is 15 (I'm fairly sure).

[swh76]

@jlashner @msilvafe be aware 15 is the max speed for ELMA and ASIS crates, but the COMTELs top out at 100.

Do be careful disabling the fan policy ; should only use to set to max or crate will no longer be able to protect hardware from overheat. On the other hand, the carriers do have some thermal protection built in ; if the FPGA temperature exceeds a (very high) threshold it will automatically shut itself down. This is the FPGA OT = over temperature protection which is built into the fw. It acts on the output of this sensor:

In [19]: S.get_fpga_temp()
Out[19]: 53.146

and the threshold can be queried like this:

In [21]: S.get_ultrascale_ot_upper_threshold()
Out[21]: 97.334

typically we have it set quite high. We may want to consider lowering it for the site systems. There is a function for setting it, https://pysmurf.readthedocs.io/en/main/client/command.html?highlight=ot#pysmurf.client.command.smurf_command.SmurfCommandMixin.set_ultrascale_ot_upper_threshold. But don't set it higher!

One thing I found recently that may be helpful ; I noticed that the SLAC ASIS crate was constantly railing despite S.get_fpga_temp() returning temperatures well below 75C (which is roughly the limit we'd like to keep the systems below, although the precise limit is a function of the FPGA current). I believe the crate controllers act on the programmed thermal thresholds. You can dump all crate sensors like this from the shelf manager;

# clia sensordata

here's an example of one of the "Temperature" sensors that the crate controller uses to decide how to set the fan speed:

88: LUN: 0, Sensor # 2 ("JunctionTemp:FPG")
    Type: Threshold (0x01), "Temperature" (0x01)
    Belongs to entity (0xa0, 0x60): FRU # 0
    Status: 0xc0
        All event messages enabled from this sensor
        Sensor scanning enabled
        Initial update completed
    Raw data: 42 (0x2a)
    Processed data: 42.000000 degrees C
    Current State Mask: 0xc0

the crate acts on preprogrammed thresholds, accessible from the shelf manager like this:

clia threshold

e.g. here's the same sensor's thresholds in my crate:

8c: LUN: 0, Sensor # 2 ("JunctionTemp:FPG")
    Type: Threshold (0x01), "Temperature" (0x01)
        Upper Non-Critical Threshold, Raw Data: 0x46 ; Processed data: 70.000000 degrees C
        Upper Critical Threshold, Raw Data: 0x64 ; Processed data: 100.000000 degrees C
        Upper Non-Recoverable Threshold, Raw Data: 0x7d ; Processed data: 125.000000 degrees C

I don't actually have a good understanding of the crate fan control logic, but it seems that if the UNCT (=Upper Non-Critical Threshold) is exceeded, the controller tries to counter by increasing the fan speed. The thresholds are preprogrammed into the boards at SLAC, but can be changed. After digging into this for a while, the sensor above I think is the source of most of our fan speed woes ; it's a different sensor for the FPGA temperature but we know from testing that it's not a very accurate sensor. In particular, it tends to read ~10-15C hotter than the more accurate sensor whose data is returned by S.get_fpga_temp() which the carrier actually uses for OT protection. After noticing this I'm finding that crates in the SLAC labs are often railing when the less accurate sensor exceeds 70C, which is set too low and systematically underestimates the FPGA temperature by 10-15 C. Adjusting this programmed threshold might allow us to run at lower fan speeds (which may be desirable because it 1) decreases wear on fans, 2) may reduce vibration from fan energy, and 3) may increase required filter replacement/cleaning interval).

One other thing on this : Jesus took a hard look at how to set the fans to run at a persistent fixed level. Find details here - https://confluence.slac.stanford.edu/pages/viewpage.action?spaceKey=ppareg&title=How+to+disable+the+FAN+policy. In particular, see instructions for how to make the change persistent. But be aware if you disable the fan policy and someone changes the fan level, the controller will not be able to counter the change. So if you make this change you'll need to make sure e.g. jackhammer doesn't set the fan speeds below max (unless it's safe to do so).

[jlashner]

This is extremely helpful. I think we can probably learn a lot by looking at JunctionTemp:FPG and the fan level over time (@msilvafe is that monitored by the smurf-crate-monitor?) I believe we were running the crate monitor for the SAT1 cooldown, so we should try to explore it a bit.

[msilvafe]

This is extremely helpful. I think we can probably learn a lot by looking at JunctionTemp:FPG and the fan level over time (@msilvafe is that monitored by the smurf-crate-monitor?) I believe we were running the crate monitor for the SAT1 cooldown, so we should try to explore it a bit.

Yup we're monitoring that and it's always been hot.

[swh76]

Uploading the SLAC internal confluence page I referenced, just in case folks don't have access.

ppareg-HowtodisabletheFANpolicy-130623-0906-10730.pdf

[jlashner]

Just wanted to post a few things that shawn brought up on slack that I think belong here.

Shawn wrote this script to regularly check temp sensors and compares them to the fan-thresholds, in order to help determine what is tripping the threshold:

https://github.com/slaclab/pysmurf/blob/main/scratch/shawn/check_crate_thermal_sensors.py

Also, max is thinking about upgrading the crate-monitor from parsing the clia sensordata output, to properly using ipmitool.

Jesus used this when writing the ATCA-monitor rogue process, so implementation details can be found here: https://github.com/slaclab/smurf-atca-monitor/blob/main/python/atcaipmi/monitor.py