Gain config not implemented for indi_gphoto_ccd / Canon 7D?

[tonycolo]

I'm experimenting with a Canon 7D. I built libgphoto 2.5.3 manually.
The camera is detected and indi_gphoto_ccd client through indi server appears to be connecting to the camera via USB cable and identify the camera as such. But I see errors from the log below.

If this should be working, I'll try writing a Python client (though I'm stuck in dependency hell with pip3 and apt-get install for libgphoto2, python-gphoto2, ekos).

Oct  2 00:39:47 astro [INFO] Capture-1/Dummy-1 indi.newMessage() #333: new Message 2023-10-02T06:39:47: [INFO] GPhoto CCD is online.
Oct  2 00:39:47 astro [INFO] Capture-1/Dummy-1 indi.newMessage() #333: new Message 2023-10-02T06:39:47: [INFO] Detected Canon Inc. Model Canon EOS 7D.
Oct  2 00:39:48 astro [INFO] Capture-1/MainThread indi.getCcdInfo() #521: Detecting bayer pattern
Oct  2 00:39:53 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception: Traceback (most recent call last):
Oct  2 00:39:53 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:   File "/home/tonycolo/sw/indi-allsky/indi_allsky/capture.py", line 176, in run
Oct  2 00:39:53 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:     self.saferun()
Oct  2 00:39:53 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:   File "/home/tonycolo/sw/indi-allsky/indi_allsky/capture.py", line 188, in saferun
Oct  2 00:39:53 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:     self._initialize()
Oct  2 00:39:53 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:   File "/home/tonycolo/sw/indi-allsky/indi_allsky/capture.py", line 589, in _initialize
Oct  2 00:39:53 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:     ccd_info = self.indiclient.getCcdInfo()
Oct  2 00:39:53 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:   File "/home/tonycolo/sw/indi-allsky/indi_allsky/camera/indi.py", line 556, in getCcdInfo
Oct  2 00:39:53 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:     gain_info = self.getCcdGain()
Oct  2 00:39:53 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:   File "/home/tonycolo/sw/indi-allsky/indi_allsky/camera/indi.py", line 1049, in getCcdGain
Oct  2 00:39:53 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:     raise Exception('Gain config not implemented for {0:s}, open an enhancement request'.format(indi_exec))
Oct  2 00:39:53 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception: Exception: Gain config not implemented for indi_gphoto_ccd, open an enhancement request
Oct  2 00:39:53 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:
Gain config not implemented for indi_gphoto_ccd, open an enhancement request

[aaronwmorris]

Merged #969 to support indi_gphoto_ccd. I had coded the support for Canon for the indi_canon_ccd server, but I see that is just an alias for indi_gphoto_ccd.

[tonycolo]

That changed things, but now after running setup.sh and choosing indi, I see:

Oct  2 19:53:24 astro [INFO] MainProcess/MainThread allsky._queueManualTasks() #1005: Checking for manually submitted tasks
Oct  2 19:53:19 astro [INFO] Capture-2/Dummy-2 indi.newMessage() #333: new Message 2023-10-03T01:53:19: [ERROR] Can not open camera: Power OK? If camera is auto-mounted as external disk storage, please unmount it and disable auto-mount. 
Oct  2 19:53:19 astro [INFO] Capture-2/Dummy-2 indi.newMessage() #333: new Message 2023-10-03T01:53:19: [ERROR] Camera open error (-105): Unknown model 
Oct  2 19:53:19 astro [WARNING] Capture-2/MainThread capture._initialize() #483: Connecting to CCD device GPhoto CCD

Note I am running indiserver manually so I can specify indi_gphoto_ccd, because:

I was "forced" to choose 'CCD Simulator' from the list of indi cameras during setup because indi_gphoto_ccd was not offered . In the left web display, the two previous cameras also show up in the drop down box, but not the gphoto one.
When choosing indi passive, I see the these errors:

Oct  2 18:14:31 astro [INFO] MainProcess/MainThread allsky._queueManualTasks() #1005: Checking for manually submitted tasks
Oct  2 18:14:44 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception: Traceback (most recent call last):
Oct  2 18:14:44 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:   File "/home/tonyb/sw/indi-allsky/indi_allsky/capture.py", line 176, in run
Oct  2 18:14:44 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:     self.saferun()
Oct  2 18:14:44 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:   File "/home/tonyb/sw/indi-allsky/indi_allsky/capture.py", line 188, in saferun
Oct  2 18:14:44 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:     self._initialize()
Oct  2 18:14:44 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:   File "/home/tonyb/sw/indi-allsky/indi_allsky/capture.py", line 429, in _initialize
Oct  2 18:14:44 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:     self.indiclient = camera_interface(
Oct  2 18:14:44 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception: TypeError: __init__() takes 10 positional arguments but 11 were given
Oct  2 18:14:44 astro [ERROR] MainProcess/MainThread allsky._startCaptureWorker() #280: Capture worker exception:

In the configuration section, it says:
You may use the config.py utility to manipulate the configuration from the command line.

So trying to inspect/set the camera somhow, after activating the virtualenv, I see:

python3 config.py list
Traceback (most recent call last):
  File "/home/tonyb/sw/indi-allsky/config.py", line 62, in <module>
    action_func(
  File "/home/tonyb/sw/indi-allsky/indi_allsky/config.py", line 558, in list
    config_list = IndiAllSkyDbConfigTable.query\
  File "/home/tonyb/sw/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/flask_sqlalchemy/model.py", line 23, in __get__
    cls, session=cls.__fsa__.session()  # type: ignore[arg-type]
  File "/home/tonyb/sw/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/orm/scoping.py", line 219, in __call__
    sess = self.registry()
  File "/home/tonyb/sw/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/util/_collections.py", line 640, in __call__
    key = self.scopefunc()
  File "/home/tonyb/sw/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/flask_sqlalchemy/session.py", line 111, in _app_ctx_id
    return id(app_ctx._get_current_object())  # type: ignore[attr-defined]
  File "/home/tonyb/sw/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/werkzeug/local.py", line 508, in _get_current_object
    raise RuntimeError(unbound_message) from None
RuntimeError: Working outside of application context.

This typically means that you attempted to use functionality that needed
the current application. To solve this, set up an application context
with app.app_context(). See the documentation for more information.

Guidance appreciated!

[aaronwmorris]

It sounds like you might have the indi programs and server split between different filesystem prefixes, like /usr/bin and /usr/local/bin. The server selection only looks at the location where it finds the indiserver binary.

You can manually edit the indiserver systemd service if you want to manually configure it.

edit ~/.config/systemd/user/indiserver.service

systemctl --user daemon-reload

systemctl --user restart indiserver

I fixed the other issues you identified in indi_passive #972 (you should not use this) and config.py list #973

I am not sure about the first error. Just make sure you are restarting the indiserver service whenever you disconnect/reconnect the camera or if it goes to sleep.

[tonycolo]

Thank you.
I reinstalled the indi_gphoto_ccd to /usr, edited the systemd service and am now using that.
config.py list now works for me.
indi_gphoto_ccd still wasn't listed in the indi camera options when rerunning setup after obtaining your latest updates, but I can select ind_gphoto_ccd from the web UI.
Somethig is amiss though:

Oct  3 23:01:12 astro [INFO] Capture-2/MainThread capture.saferun() #210: Exposure state: ALERT
Oct  3 23:01:12 astro [INFO] Capture-2/MainThread capture.saferun() #209: Camera last ready: 22.1s
Oct  3 23:01:11 astro [WARNING] Capture-2/MainThread capture.sigterm_handler_worker() #149: Caught TERM signal
Oct  3 23:01:11 astro [INFO] MainProcess/MainThread allsky._stopCaptureWorker() #324: Stopping Capture worker
Oct  3 23:01:11 astro [INFO] MainProcess/MainThread allsky._stopCaptureWorker() #321: Terminating Capture worker
Oct  3 23:01:11 astro [WARNING] MainProcess/MainThread allsky.run() #557: Shutting down
Oct  3 23:01:01 astro [INFO] Capture-2/MainThread indi.getCcdTemperature() #857: Sensor temperature: 0.0
Oct  3 23:01:01 astro [INFO] Capture-2/MainThread capture.detectMoonMode() #1302: Moon Mode conditions detected
Oct  3 23:01:01 astro [INFO] Capture-2/MainThread capture.detectMoonMode() #1298: Moon altitude: 16:33:00.2, phase 73.1%
Oct  3 23:01:01 astro [INFO] Capture-2/MainThread capture.detectNight() #1280: Sun altitude: -47:40:16.7
Oct  3 23:01:01 astro [INFO] Capture-2/MainThread capture.saferun() #210: Exposure state: ALERT
Oct  3 23:01:01 astro [INFO] Capture-2/MainThread capture.saferun() #209: Camera last ready: 11.0s
Oct  3 23:00:59 astro [WARNING] Video-1/MainThread video.sigterm_handler_worker() #120: Caught TERM signal
Oct  3 23:00:59 astro [WARNING] Image-1/MainThread image.sigterm_handler_worker() #204: Caught TERM signal
Oct  3 23:00:59 astro [WARNING] Capture-2/MainThread capture.sigterm_handler_worker() #149: Caught TERM signal
Oct  3 23:00:58 astro [WARNING] MainProcess/MainThread allsky.sigterm_handler_main() #189: Caught TERM signal, shutting down
Oct  3 23:00:58 astro [WARNING] MainProcess/MainThread allsky.sigterm_handler_main() #189: Caught TERM signal, shutting down
Oct  3 23:00:58 astro [INFO] MainProcess/MainThread allsky._queueManualTasks() #1005: Checking for manually submitted tasks
Oct  3 23:00:51 astro [INFO] Capture-2/Dummy-2 indi.newMessage() #333: new Message 2023-10-04T05:00:51: [INFO] Please update the CCD Information in the Image Info section before proceeding. The camera resolution shall be updated after the first exposure is complete. 
Oct  3 23:00:51 astro [INFO] Capture-2/MainThread capture.saferun() #416: Total time since last exposure 0.1625 s
Oct  3 23:00:51 astro [INFO] Capture-2/MainThread indi.set_number() #1239: Setting Duration (s) = 0.0010000099427998066
Oct  3 23:00:51 astro [INFO] Capture-2/MainThread capture.shoot() #1446: Taking 0.00100001 s exposure (gain 100)
Oct  3 23:00:51 astro [WARNING] Capture-2/MainThread indi.setCcdBinning() #1182: indi_gphoto_ccd does not support bin settings
Oct  3 23:00:51 astro [WARNING] Capture-2/MainThread indi.setCcdBinning() #1150: Setting CCD binning to (1, 1)
Oct  3 23:00:50 astro [INFO] Capture-2/Dummy-2 indi.newMessage() #333: new Message 2023-10-04T05:00:50: Error: invalid state switch for property ISO (CCD_ISO). No switch is on. 
Oct  3 23:00:50 astro [INFO] Capture-2/MainThread indi.configureDevice() #699: Setting switch CCD_ISO
Oct  3 23:00:50 astro [ERROR] Capture-2/MainThread indi.setCcdGain() #1108: Canon ISO not found for 100, using ISO 100
Oct  3 23:00:50 astro [INFO] Capture-2/MainThread indi.setCcdGain() #1102: Mapping gain to ISO for libgphoto device
Oct  3 23:00:50 astro [WARNING] Capture-2/MainThread indi.setCcdGain() #1064: Setting CCD gain to 100
Oct  3 23:00:50 astro [WARNING] Capture-2/MainThread capture.reconfigureCcd() #1246: Change to night (moon mode)
Oct  3 23:00:50 astro [INFO] Capture-2/MainThread indi.getCcdTemperature() #857: Sensor temperature: 0.0
Oct  3 23:00:50 astro [INFO] Capture-2/MainThread capture.detectMoonMode() #1302: Moon Mode conditions detected
Oct  3 23:00:50 astro [INFO] Capture-2/MainThread capture.detectMoonMode() #1298: Moon altitude: 16:31:01.8, phase 73.1%
Oct  3 23:00:50 astro [INFO] Capture-2/MainThread capture.detectNight() #1280: Sun altitude: -47:38:49.5
Oct  3 23:00:50 astro [INFO] Capture-2/MainThread capture.saferun() #210: Exposure state: unset
Oct  3 23:00:50 astro [INFO] Capture-2/MainThread capture.saferun() #209: Camera last ready: 0.0s
Oct  3 23:00:47 astro [ERROR] Capture-2/MainThread capture._initialize() #773: CCD day gain below minimum, changing to 100
Oct  3 23:00:45 astro [INFO] MainProcess/MainThread allsky._queueManualTasks() #1005: Checking for manually submitted tasks
Oct  3 23:00:43 astro [ERROR] Capture-2/MainThread capture._initialize() #764: CCD moon mode gain below minimum, changing to 100
Oct  3 23:00:43 astro [INFO] Capture-2/MainThread capture._initialize() #747: Default CCD exposure: 0.00100001
Oct  3 23:00:43 astro [INFO] Capture-2/MainThread capture._initialize() #703: Maximum CCD exposure: 25.00000000
Oct  3 23:00:43 astro [INFO] Capture-2/MainThread capture._initialize() #682: Minimum CCD exposure: 0.00100001
Oct  3 23:00:42 astro [INFO] Capture-2/MainThread indi.set_switch() #1267: Enabling Light (FRAME_LIGHT)
Oct  3 23:00:42 astro [INFO] Capture-2/MainThread indi.configureDevice() #699: Setting switch CCD_FRAME_TYPE
Oct  3 23:00:42 astro [INFO] Capture-2/MainThread indi.updateCcdBlobMode() #398: Set BLOB mode
Oct  3 23:00:41 astro [INFO] Capture-2/MainThread indi.set_switch() #1267: Enabling Disable (DISABLE)
Oct  3 23:00:41 astro [INFO] Capture-2/MainThread indi.configureDevice() #699: Setting switch DEBUG
Oct  3 23:00:41 astro [INFO] Capture-2/MainThread miscDb.addCamera() #83: Camera DB ID: 4
Oct  3 23:00:41 astro [INFO] Capture-2/MainThread indi.getCcdInfo() #521: Detecting bayer pattern
Oct  3 23:00:40 astro [INFO] Capture-2/Dummy-2 indi.newMessage() #333: new Message 2023-10-04T05:00:40: [INFO] Detected Canon Inc. Model Canon EOS 7D. 
Oct  3 23:00:40 astro [INFO] Capture-2/Dummy-2 indi.newMessage() #333: new Message 2023-10-04T05:00:40: [INFO] GPhoto CCD is online. 
Oct  3 23:00:40 astro [WARNING] Capture-2/MainThread capture._initialize() #483: Connecting to CCD device GPhoto CCD

The camera is in Bulb mode and there is no internal SD card.
And the https://astro.local/indi-allsky/camera
page only says "Internal error"

[aaronwmorris]

Can you check /var/log/indi-allsky/webapp-indi-allsky.log for the reason for the internal error.

Unfortunately, I am in the middle of moving to a new home, so I do not have my Canon camera to check to see if something has changed with a new version of indi. After trying to run indi-allsky, can you paste the output of indi_getprop here?

[tonycolo]

Understood about moving and no camera--appreciate the help to date.

That log file only has repeated lines of:

Oct 4 00:22:06 astro [WARNING] MainProcess views.dispatch_request() #3889: indi-allsky log empty

The output of indi_getprop.

[tonycolo]

I believe part of the problem is incompatible libraries due to the way I built various libraries.

So I decided to start from scratch with a fresh instance of Raspbian Bookworm 64-but.

Unfortunately, setup.sh warned me about indi 2.0.4 not being compatible with pyindi-client and the install fails per below.
I had only installed gphoto2 and libgphoto2 before running setup.

ERROR: Failed building wheel for pyindi-client
  Running setup.py clean for pyindi-client
Failed to build pyindi-client
ERROR: Could not build wheels for pyindi-client, which is required to install pyproject.toml-based projects

[aaronwmorris]

The fix for indi 2.0.4 was included in the URL in the warning.

https://github.com/aaronwmorris/indi-allsky/wiki/INDI-2.0.4-bug

[tonycolo]

Editing the .h file allowed setup to finish. I am wondering if there should be a link to the URL in the setup dialog that warns about this so that people don't have to hunt for the URL you provide.

In any event, I am able to take pictures with gphoto2, but indi-allsky never takes a picture. The only remaining information in the log are Alerts about exposure state:

2023-10-23T00:32:40.534087-06:00 astro [INFO] Capture-1-40689/MainThread capture.saferun() #210: Exposure state: ALERT

And the "internal error" response for https://astro.local/indi-allsky/camera remains.

[aaronwmorris]

There is a URL in the 2.0.4 warning in the latest code.

I will try to get my Canon camera out in the next couple of days to test.

[tonycolo]

I should have been more specific - I didn't see a hint how to fix in the quasi-GUI blue dialog boxes when it warned about 2.0.4.

[aaronwmorris]

I am not sure if you have updated to the latest code. I had tweaked the language a bit to indicate that there is a fix.

[tonycolo]

I have updated to the latest and rerun setup.
I now see the 2.0.4 warning, but the same ALERT behavior is resulting.

I experimented with the CCD Tutorial in pyindi-client to try to use gphoto_ccd:

ccd = "GPhoto CCD" #"CCD Simulator"
device_ccd = indiclient.getDevice(ccd)

When I ask for the state right before the exposure loop it's PyIndi.IPS_IDLE (i.e. 0).
But in the main exposure loop, it blocks on blobEvent.wait()

This is with the 7D in M mode.
I am wondering if there is some incompatibility between indi 2.0.3/4, pyindi-client and gphoto_ccd.

[aaronwmorris]

Reading back through the comments, I think you may need to define the CCD_INFO properties for the camera, as this information is not auto-detected. If these are not defined, indi might not function correctly.

I tested with my Rebel T6 (1300D) and it functioned fine with INDI 2.0.4.

Oct  3 23:00:51 astro [INFO] Capture-2/Dummy-2 indi.newMessage() #333: new Message 2023-10-04T05:00:51: [INFO] Please update the CCD Information in the Image Info section before proceeding. The camera resolution shall be updated after the first exposure is complete. 

From: https://github.com/aaronwmorris/indi-allsky/wiki/INDI-custom-config#dslr

{
    "PROPERTIES" : {
        "CCD_INFO" : {
            "CCD_MAX_X" : 5184,
            "CCD_MAX_Y" : 3456,
            "CCD_PIXEL_SIZE" : 4.3,
            "CCD_PIXEL_SIZE_X" : 4.3,
            "CCD_PIXEL_SIZE_Y" : 4.3,
            "CCD_BITSPERPIXEL" : 16
        }
    },
    "SWITCHES": {
        "CCD_CAPTURE_FORMAT": {
            "on": [
                "FORMAT_7"
            ]
        }
    }
}

Note: Based on your indi_getprop output, I think FORMAT_7 is the correct for RAW on the 7D

[tonycolo]

I did not realize that in the web UI under Config, I could plug that JSON into the INDI Camera Configuration.
As you surmised, this was the problem and now exposures are happening. Thank you.

[aaronwmorris]

Fantastic! Glad to hear.

[tonycolo]

On to the next problem with the 7D: the images are being post-processed and "overdriven", at least I believe, by allsky to be entirely uniformly gray. I've tried changing in the camera tab the night and day gain modes, but to no avail. Any suggestions?

[aaronwmorris]

Try setting the camera bit depth to 14-bits (instead of "Auto Detect"). The other option is setting a default exposure of something higher like 0.1s.

At low exposures, indi-allsky relies on noise from the image to determine the dynamic range. At low exposures/gain, there is not always enough noise and indi-allsky will assume it is a lower bit depth (eg 10 or 12 bits) when it downsamples to 8-bits, and this results in a grey image.