Changing the image filenames to incorporate a timestamp

[KMBlundell]

Hi,

I need some help in upgrading from an old version to the latest version of the allsky package. I've been using the system since version 0.6 but it seemed worthwhile to update as there have been some very useful changes made. I have installed 0.8.3.3 on a new Pi4 and have everything working perfectly, so there is no problem with standard operations - my problem is that I have been running 0.6 (and 0.7) with a small tweak, and think that now there may be a better way to do this with the recent improvements to the system. Let me explain what I've been doing and why (sorry if this is a bit long!).

I have the Pi on the same (secure) network as the observing (Windows) computer and set them up with a shared directory. By simply copying the current image into this shared space on the Pi allowed the observing computer to see (and archive) the latest image. (This avoids the unnecessary step of setting up an FTP server.) The latest image was given a unique name by appending the date and time to a base name. Irfanview was configured to "watch" the directory and automatically display any new image. This worked perfectly - except it required adding a line into one of the scripts, so it wasn't exactly a portable configuration change and required effort to maintain when changing versions. The additional line was:

cp $CURRENT_IMAGE "Windows-share/$FILENAME-$(date+'%Y-%m-%d_%H-%M-%S').$EXTENSION"

I thought that I could use the upload option in the new system now that it has a local option intended for updating a local web server. But this only copies image.jpg and overwrites it each time, which is not what I want. Is there a way to modify the copied filename by appending the date and time - in the same way that movies, startrails & keograms can be renamed (parameter in ftp-settings.sh - although I think this only supports a static name)? Is this a new feature request or is there a way to do it already?

At the moment, I have modified upload.sh to break DESTINATION_FILE into the filename and extension and then add the date & time back into a modified DESTINATION_FILE to give a unique name - but I can't get the hyphens in the date to appear as I did before, so the date and time string just becomes a continuous number sequence. I don't know if this because the new Pi is running bullseye and the old one is running buster and there's been some weird change, or whether my bash shell programming skills are not up to the job (most likely). The following lines are what I've inserted and don't quite work - can somebody please tell me why the hyphens don't appear in the filename:

REMOTE_DIR="${2}"
DF="${3}"
EXT="${DF##*.}"
FN="${DF%.*}"
DESTINATION_FILE="$(FN-$(date +'%Y-%m-%d_%H-%M-%S').$EXT)"

I note that the images saved in the dated directories don't have hyphens between the fields either. What governs this naming convention and can it be changed?

In going through the scripts to try to follow what is happening I came across this apparent duplication which I don't understand. The following are selected lines from the saveimage.sh script:

113 SAVED_FILE="${CURRENT_IMAGE}" # The name of the file saved from the camera.
114 WEBSITE_FILE="${WORKING_DIR}/${FULL_FILENAME}" # The name of the file the websites look for

159 if [ "${IMG_UPLOAD}" = "true" ] ; then

181 # We didn't create ${WEBSITE_FILE} yet so do that now.
182 mv "${CURRENT_IMAGE}" "${WEBSITE_FILE}"

210 fi
212 # We create $WEBSITE_FILE as late as possible to avoid it being overwritten.
213 mv "${SAVED_FILE}" "${WEBSITE_FILE}"
214
215 exit 0

Line 182 (executed only if upload=true) appears to be identiical to line 213 which is always executed.

I also don't understand why the same file is being given multiple identities. I find this very confusing when trying to understand what is going on.

It would be greatly appreciated if somebody can answer these questions. Thanks.

[EricClaeys]

@KMBlundell Hi, I never programmed in bash before helping with Allsky, although I programmed in sh and ksh (which is very similar to bash).

The reason dashes aren't being added to the date is most likely this line:

DESTINATION_FILE="$(FN-$(date +'%Y-%m-%d_%H-%M-%S').$EXT)"

The date part is fine but $(FN....) isn't. In bash, $(cmd) executes cmd in a subshell and returns the results. You want ${FN}- with no ) at the end. I suspect if you look in /var/log/allsky.log you'll see an error message for this line.

---

Re: I note that the images saved in the dated directories don't have hyphens between the fields either. What governs this naming convention and can it be changed?
The allsky/src/capture_*.cpp files do that. In theory it could be changed but would require changing several other programs that assume the format of the names (and directories in allsky/image) have no hyphens. Can't Infraview handle files without hyphens?

---

A basic assumption in Allsky is that whatever displays images only displays the current image. This is true in the Live View of the WebUI and the Allsky Website, be it on the Pi or a remote server. This is why there's no way to upload a file with a non-static name.
It sounds like in your case this assumption doesn't hold, and you want the "observing computer" (I assume the Windows machine) to see and archive the latest image. Do you have two copies of every image, one on the Pi and one on Windows? Does the PC let you view any archived image?
However, I think I can implement what you want by changing ${FULL_NAME} (image.jpg) in this line in saveImage.sh:

"${ALLSKY_SCRIPTS}/upload.sh" "${FILE_TO_UPLOAD}" "${IMAGE_DIR}" "${FULL_FILENAME}" "SaveImage" "${WEB_IMAGE_DIR}"

to $(basename "${FILE_TO_UPLOAD}"). This would require a setting, probably in ftp-settings.sh that specifies to use the original name or ${FULL_FILENAME}. In your case you'd use the "local" PROTOCOL and specify the directory on the Pi where the file should be copied to. You'd need to change your end to accept file names without dashes.

Would this work for your?

---

When an image is first saved to disk by the capture programs, it has a name of allsky/tmp/image-YYYYMMDDHHMMSS.jpg which is the same as the name the image will have in allsky/images/DATE. However, the WebUI, Allsky Website, and most other websites display only the current image and need to know its name. It would be difficult for them to look for the newest image-YYYYMMDDHHMMSS.jpg file and display it, so we use a generic name that never changes, i.e., image.jpg. We do all the processing (resize/crop/stretch) of image-YYYYMMDDHHMMSS.jpg in allsky/tmp, then if the image is to be saved, we execute:

cp "${CURRENT_IMAGE}" "${FINAL_FILE}" || echo "*** ERROR: ${ME}: unable to copy ${CURRENT_IMAGE} ***"

where ${FINAL_FILE} is in allsky/image/YYYYMMDD. After that, ${CURRENT_IMAGE} is renamed to ${WEBSITE_FILE} (which is image.jpg).

This statement:

181 # We didn't create ${WEBSITE_FILE} yet so do that now.
182 mv "${CURRENT_IMAGE}" "${WEBSITE_FILE}"

is in a block of code that implements the ability to to only upload every Nth image, for example, every 3rd image. Line 182 is executed if the CURRENT_IMAGE should NOT be uploaded. However, it still needs to be renamed (to "image.jpg") so the WebUI and Allsky Website can find it. Note that line 184 does exit 0 which exits saveImage.sh.
The "mv" on line 213 is executed for every image EXCEPT the ones that shouldn't be uploaded since they were already renamed.

Does that make sense?

[KMBlundell]

Yes! Thank you very much Eric, for answering my questions.

The reason for wanting the formatted date is that at some point a human may look at the files and a 14-digit numerical string is hard to decode at a glance, while inserting some formatting in the right places helps considerably.   All the various astronomical files I process are created with this formatting in their filename and I wanted to keep the same convention for the all-sky camera images too.

Yes, there are 2 copies of the files - at least for a short time.   We don't regard a Pi as an archiving computer, so those files are automatically deleted after a week, but the Windows PC files are actually then archived onto a larger server along with the other data taken during the night.

If I'm the only person wanting this formatting in the filename, plus wanting dynamic file names, then there is little point in forcing the change.   It is trivial to insert the line I previously used into saveImage.sh, which is what I've now done and it works fine.   (I had hoped that with all the changes that have happened recently that the "local" upload system could accommodate this.)    The problem with this is that the date & time string which I use is for when the file is processed, which is a few seconds after the proper time of the image readout, so I now need to brush up even more on my bash scripting to see if I can format the string extracted from the filename rather than inserting a new string which is out by a few seconds.

[EricClaeys]

@KMBlundell, I'll implement the ability to have the uploaded file retain its YYYYMMDDHHMMSS name since it's fairly simple. It will be in the next release which we're almost ready for user testing.
It won't however, add dashes in the names - that's too much effort and likely will break other people's code. I agree the names are much easier to read with dashes or dots or something - I've struggled with that myself. I think it's one of those things that if Thomas started over, would be done differently.