Error in image processing #695

czimm74 · 2023-04-18T17:32:37Z

czimm74
Apr 18, 2023

I am not sure when, but sometime during the night something in the back-end went haywire. It is taking photos and storing them properly (thought maybe the storage failed but nothing indicates that). I have tried resetting the services and reboots of the Pi. But to no avail.

I am not sure when this started, last I had looked at it the night before all was fine.

Here is what I am seeing the in the log screen:

Apr 18 13:11:09 indi-allsky [INFO] MainProcess allsky._startImageWorker() #771: Starting ImageWorker process 7
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception:
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: (Background on this error at: https://sqlalche.me/e/20/4xp6)
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: [parameters: ('/media/pi/USB_Pi/allsky/images/ccd_ce4df499-9a63-4e9d-8370-0172c5a771ea/20230418/day/18_13/ccd1_20230418_131100.jpg', None, None, '2023-04-18 13:11:00.435833', '2023-04-18', 0.0010000099427998066, 2.207105875015259, 1.4963407516479492, 0, 1, None, 0, 163.65215220815435, 0, 0, None, 0, 3230.8844567218443, 0, None, 0, 0, 1)]
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: [SQL: INSERT INTO image (filename, remote_url, s3_key, "createDate", "dayDate", exposure, exp_elapsed, process_elapsed, gain, binmode, "temp", night, adu, stable, moonmode, moonphase, adu_roi, sqm, stars, sync_id, calibrated, detections, camera_id) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: sqlalchemy.exc.DatabaseError: (sqlite3.DatabaseError) database disk image is malformed
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: cursor.execute(statement, parameters)
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 748, in do_execute
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: self.dialect.do_execute(
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1964, in _exec_single_context
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2326, in _handle_dbapi_exception
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: self._handle_dbapi_exception(
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1983, in _exec_single_context
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: return self._exec_single_context(
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1842, in _execute_context
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: ret = self._execute_context(
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1638, in _execute_clauseelement
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: return connection._execute_clauseelement(
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 486, in _execute_on_connection
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: return meth(
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1414, in execute
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: result = connection.execute(
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1184, in _emit_insert_statements
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: _emit_insert_statements(
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: util.preloaded.orm_persistence.save_obj(
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 644, in execute
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: rec.execute(self)
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 467, in execute
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: flush_context.execute()
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 4252, in _flush
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: raise exc_value.with_traceback(exc_tb)
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 147, in exit
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: transaction.rollback(_capture_exception=True)
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 4292, in _flush
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: self._flush(objects)
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 4155, in flush
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: self.session.flush()
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1193, in _prepare_impl
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: ret_value = fn(self, *arg, **kw)
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 137, in _go
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "", line 2, in _prepare_impl
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: self._prepare_impl()
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1218, in commit
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: ret_value = fn(self, *arg, **kw)
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 137, in _go
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "", line 2, in commit
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: trans.commit(_to_root=True)
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1903, in commit
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: return self._proxied.commit()
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/orm/scoping.py", line 553, in commit
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: db.session.commit()
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/indi_allsky/flask/miscDb.py", line 231, in addImage
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: image_entry = self._miscDb.addImage(
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/indi_allsky/image.py", line 461, in processImage
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: self.processImage(i_dict)
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/indi_allsky/image.py", line 235, in saferun
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: self.saferun()
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/indi_allsky/image.py", line 206, in run
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: Traceback (most recent call last):
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception:
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: The above exception was the direct cause of the following exception:
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception:
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: sqlite3.DatabaseError: database disk image is malformed
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: cursor.execute(statement, parameters)
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 748, in do_execute
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: self.dialect.do_execute(
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: File "/home/pi/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1964, in _exec_single_context
Apr 18 13:11:09 indi-allsky [ERROR] MainProcess allsky._startImageWorker() #764: Image worker exception: Traceback (most recent call last):
Apr 18 13:11:09 indi-allsky [INFO] MainProcess allsky.run() #1039: Exposure state: READY
Apr 18 13:11:09 indi-allsky [INFO] MainProcess allsky.run() #1038: Camera last ready: 8.7s
Apr 18 13:11:02 indi-allsky [INFO] ImageWorker006 miscDb.addImage() #171: Adding image /media/pi/USB_Pi/allsky/images/ccd_ce4df499-9a63-4e9d-8370-0172c5a771ea/20230418/day/18_13/ccd1_20230418_131100.jpg to DB
Apr 18 13:11:02 indi-allsky [INFO] ImageWorker006 image.write_img() #1084: Image compressed in 0.1252 s
Apr 18 13:11:01 indi-allsky [INFO] ImageWorker006 image.processImage() #429: Image processed in 1.4963 s
Apr 18 13:11:01 indi-allsky [INFO] ImageWorker006 image.apply_logo_overlay() #2332: Alpha transparency in 0.2606 s
Apr 18 13:11:01 indi-allsky [INFO] ImageWorker006 image.apply_image_circle_mask() #2307: Image circle mask in 0.0931 s
Apr 18 13:11:00 indi-allsky [INFO] ImageWorker006 image.white_balance_manual_bgr() #2157: Applying manual color balance settings
Apr 18 13:11:00 indi-allsky [WARNING] ImageWorker006 image.recalculate_exposure() #1312: New calculated exposure: 0.001000
Apr 18 13:11:00 indi-allsky [INFO] ImageWorker006 image.calculate_histogram() #1216: Brightness average: 163.65
Apr 18 13:11:00 indi-allsky [INFO] ImageWorker006 image.processImage() #323: Image: 2028 x 1520
Apr 18 13:11:00 indi-allsky [INFO] ImageWorker006 sqm.calculate() #42: Weighted SQM average: 3230.88
Apr 18 13:11:00 indi-allsky [INFO] ImageWorker006 sqm.calculate() #37: Raw SQM average: 183.58
Apr 18 13:11:00 indi-allsky [WARNING] ImageWorker006 sqm._generateSqmMask() #63: Using central 20% ROI for SQM calculations
Apr 18 13:11:00 indi-allsky [INFO] ImageWorker006 sqm._generateSqmMask() #48: Generating mask based on SQM_ROI
Apr 18 13:11:00 indi-allsky [INFO] ImageWorker006 sqm.calculate() #21: Exposure: 0.001000, gain: 0
Apr 18 13:11:00 indi-allsky [WARNING] ImageWorker006 image._calibrate() #1850: Dark not found: ccd1 8bit 0.0010000s gain 0 bin 1 0.00c
Apr 18 13:11:00 indi-allsky [WARNING] ImageWorker006 image._calibrate() #1832: Temperature matched dark not found: 0.00c
Apr 18 13:11:00 indi-allsky [INFO] ImageWorker006 image._calibrate() #1815: Searching for dark frame: gain 0, exposure >= 0.0, temp >= 0.0c
Apr 18 13:11:00 indi-allsky [WARNING] ImageWorker006 image._calibrate() #1803: Bad Pixel Map not found: ccd1 8bit 0.0010000s gain 0 bin 1 0.00c
Apr 18 13:11:00 indi-allsky [WARNING] ImageWorker006 image._calibrate() #1785: Temperature matched bad pixel map not found: 0.00c
Apr 18 13:11:00 indi-allsky [INFO] ImageWorker006 image._calibrate() #1768: Searching for bad pixel map: gain 0, exposure >= 0.0, temp >= 0.0c
Apr 18 13:11:00 indi-allsky [INFO] ImageWorker006 image._detectBitDepth() #1712: Image max value: 222
Apr 18 13:11:00 indi-allsky [INFO] ImageWorker006 image.add() #1659: Image bits: 8, cfa: None
Apr 18 13:11:00 indi-allsky [INFO] MainProcess allsky.run() #1180: Exposure received in 2.2098 s (2.2088)
Apr 18 13:10:58 indi-allsky [INFO] MainProcess allsky.run() #1272: Total time since last exposure 120.0112 s
Apr 18 13:10:58 indi-allsky [INFO] MainProcess libcamera.setCcdExposure() #192: image command: libcamera-still --immediate --nopreview --encoding jpg --quality 100 --gain 0 --shutter 1000 --metadata /tmp/tmp4d42skfw.json --metadata-format json --height 1520 --width 2028 --tuning-file /usr/share/libcamera/ipa/raspberrypi/imx477_noir.json --output /tmp/tmp1xb1o7xg.jpg
Apr 18 13:10:58 indi-allsky [INFO] MainProcess allsky.shoot() #1824: Taking 0.00100001 s exposure (gain 0)
Apr 18 13:10:58 indi-allsky [INFO] MainProcess allsky._queueManualTasks() #2422: Checking for manually submitted tasks

aaronwmorris · 2023-04-19T22:24:10Z

aaronwmorris
Apr 19, 2023
Maintainer

The SQLite database is corrupt. Shutdown indi-allsky and see if an integrity check will fix it.

sqlite3 /var/lib/indi-allsky/indi-allsky.sqlite "PRAGMA integrity_check;"

0 replies

aaronwmorris · 2023-04-30T17:32:44Z

aaronwmorris
Apr 30, 2023
Maintainer

I have also merged #725 to help prevent database corruption.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error in image processing #695

{{title}}

Replies: 2 comments

{{title}}

{{title}}

Select a reply

Error in image processing #695

czimm74 Apr 18, 2023

Replies: 2 comments

aaronwmorris Apr 19, 2023 Maintainer

aaronwmorris Apr 30, 2023 Maintainer

czimm74
Apr 18, 2023

aaronwmorris
Apr 19, 2023
Maintainer

aaronwmorris
Apr 30, 2023
Maintainer