Allsky FTP Upload error

[jlfutari]

Hi All,
When I use the command -> cd ~/allsky; website/install.sh --remote , I get this error -> "cd: Fatal error: gnutls_handshake: An unexpected TLS packet was received." (twice).

I'm not sure how to fix/resolve this FTP error.

Any help would be great.

System
Rasp PI 4
Rasp PI OS (Bullseye)
Allsky Version: v2023.05.01_04

(screen shot attached)
James

[EricClaeys]

I suggest upgrading to the newest Allsky release, v2024.12.06. It has a new command to test an upload and in many cases, determine why it failed.
It also has simplified the process for installing and configuring remote Websites.
Overall, it's easier to troubleshoot.

See te main GitHub page for a link to installation instructions.

[EricClaeys]

@jlfutari, any update?

[EricClaeys]

@jlfutari, I googled the error. It worked for one user when they used set ftp:ssl-auth TLS.
Try adding that to LFTP_COMMANDS.

[EricClaeys]

Is this a new remote Website? If not, what changed?

Someone else used set ssl:check-hostname false

[jlfutari]

Hi EricClaeys,

Sorry for my lack of reply's and thankyou for suggestions above

I have upgraded to the latest version of allsky on my Rasp PI. But other than that I haven't had time to continue with the (new)website setup (life is to busy).

Fingers crossed I will get to it this week.

James

[jlfutari]

Hi EricCleays,

I've had a few minutes today to read throw the posts in this thread and I haven't had much joy.

Of note, I also have a AVM FritzBox Modem(same as Radinkel)......

I tried using - set ssl:check-hostname false ; set ftp:ssl-auth TLS - the FTP Commands setting and got the following output in allsky (see attached screen shot). I've also inluded a second screen shot from the command line in RaspPI OS

James

[Radinkel]

Do you still get the handshake-error if you switch to FTP for testing? For me it works in principle, but extremely slowly. A possibly certificate-error can be avoided by set ssl:verify-certificate no
Ralf

[EricClaeys]

@jlfutari, @Radinkel,
I'm sure you both have the same problem given that you are getting identical messages and using the same service.
Are you both just starting to use an Allsky Website? If not and it used to work, then something changed at your provider.
I use 1and1 / IONOS which is also a big company and they have been responsive when I had questions. Does your provider have an on-line help page? It might explain how to configure an FTP client ( 1and1 does).

[Radinkel]

@EricClaeys
Yes, for me its first installation.
I asked the developer of the allskeye software, with which I also had an FTP upload problem to the Fritz-NAS: But his problem related to "Cannot read remote file" (for testing purpose): It looks like a bug in the FTP server.
With SIZE command it responds with the file size, but with MDTM command with the very same path it claims that the file does not exist.
You should be able to workaround this by forcing WinSCP to use LIST command instead of MDTM using:
sessionOptions.AddRawSettings("ProtocolFeatures", "-SIZE,-MDTM");
So thats nothing to do with our recent problems.

And I asked the AVM-Forum. I couldn't find anything relevant to our problem there
Ralf

[jlfutari]

@EricClaeys - Yes, this is my first time using the Allsky website

[Radinkel]

Hi EricCleays,
I got exactly the same error.
Test upload reports:

• address found
• connecting to .....
• **** gnutls_handshake: An unexpected TLS packet was received.

I tied set ssl:verify-certificate no (because at the beginning I got a error related to invalid certificate), set ssl:check-hostname false and set ftp:ssl-auth TLS in various combinations.
Do I have to set the commands into 'xxxx' and seperated by :?
e.g. 'set ssl:check-hostname false':'set ftp:ssl-auth TLS'

I use V2024-12-06 and a FTP upload to the popular (in Germany) Fritzbox. Surprisingly I got 1 image uploaded, then nothing.

Over all the configuration of the new version is much more straightforward!
Ralf

[EricClaeys]

@Radinkel, Ralf,
Enter all the ftp commands into the FTP Commands setting, separating commands with a ;. Do not use any quotes.

Then run testUpload.sh --debug --website and let me know how it goes.

[Radinkel]

Very funny:
when I set to ftp-upload, I get the invalid certificate error, which can be fixed with the set ssl:veryfy-certificate no command (without ' at beginning and end): Test upload to Remote Server succeeded. In correspondig folder I get sometimes a SaveImage-server-xxxx, somtimes an image.jpg. I can download and rename into a *.jpg and open, sometimes a complete image, sometimes incomplete....
I will observe it...
Ralf

For ftps-upload I get the handshake error I can't get rid of.

[EricClaeys]

You can safely remove all the SaveImage file that are older than a couple minutes.
When a file is uploaded it is given a temporary name like SaveImage-12345, then renamed to the permanent name like image.jpg.
For some reason the rename fails so the temporary file is left. This has been happening for years. I tried to figure it out once but couldn't so I gave up. A future Allsky release will automatically remove those files.

[Radinkel]

Eric,
sorry for the delay. I'm not at home at Linux-world....
I get this, after a Minute...

admin@allsky:~/allsky/scripts $ ./testUpload.sh --debug --server

See output

Executing:
/home/admin/allsky/scripts/upload.sh --debug --remote-server /tmp/testUpload.sh.txt Spica/allsky/ testUpload.sh.txt testUpload.sh
Test upload to Remote Server succeeded.
OUTPUT:
Upload output from '/tmp/testUpload.sh.txt:
---- Resolving host address...
---- 2 addresses found: 2001:9e8:202d:bf6f:9a9b:cbff:fe6f:27f5, 92.117.244.67
---- Connecting to ehcj2i7etqqinlyu.myfritz.net (2001:9e8:202d:bf6f:9a9b:cbff:fe6f:27f5) port 42817
<--- 220 FTP server ready.
---> FEAT
<--- 211- Extensions supported:
<--- UTF8
<--- MDTM
<--- SIZE
<--- AUTH TLS
<--- PBSZ
<--- PROT
<--- 211 end
---> AUTH TLS
<--- 234 Authentication method accepted
---> OPTS UTF8 ON
WARNING: Certificate verification: The certificate is NOT trusted. The certificate issuer is unknown. (BF:D0:CE:F9:08:7F:9A:2F:66:F1:EF:60:AA:1C:FB:C5:A2:8D:7D:4C)
<--- 530 Please login with USER and PASS.
---> USER radinkel
<--- 331 Password required for radinkel.
---> PASS XXXX
<--- 230 User radinkel logged in.
---> OPTS UTF8 ON
<--- 200 UTF8 ON
---> PWD
<--- 257 "/" is current directory.
---> PBSZ 0
<--- 200 ok
---> PROT P
<--- 200 Data channel will be secured
---> TYPE I
<--- 200 Type set to I.
---> PWD
257 "/" is current directory.
---> TYPE A
<--- 200 Type set to A.
---> EPSV
<--- 229 Entering Extended Passive Mode (|||41383|)
---- Connecting data socket to (2001:9e8:202d:bf6f:9a9b:cbff:fe6f:27f5) port 41383
---> LIST
<--- 150 Opening ASCII mode data connection for '/bin/ls -lgA'.
WARNING: Certificate verification: The certificate is NOT trusted. The certificate issuer is unknown. (BF:D0:CE:F9:08:7F:9A:2F:66:F1:EF:60:AA:1C:FB:C5:A2:8D:7D:4C)
drwxrwxr-x 2 ftp ftp 240 Dec 16 2011 Bilder
drwxrwxr-x 2 ftp ftp 312 Dec 16 2011 Dokumente
drwxrwxrwx 5 ftp ftp 440 Feb 1 2023 FRITZ
drwxrwxr-x 2 ftp ftp 232 Dec 16 2011 Musik
drwxrwxrwx 4 ftp ftp 296 Jan 6 21:17 Spica
drwxrwxr-x 2 ftp ftp 312 Dec 16 2011 Videos
drwxrwxrwx 5 ftp ftp 368 Jan 6 20:04 allsky
-rwxrw-rw- 1 ftp ftp 116 Feb 1 2024 copy_keogram.bat
<--- 226 Transfer complete.
-rw-rw-rw- 1 ftp ftp 8 Aug 29 2024 ftp_test.txt
---> CWD Spica
<--- 250 CWD command successful.
---> CWD allsky
<--- 250 CWD command successful.
In DIRECTORY=Spica/allsky/:
---> EPSV
<--- 229 Entering Extended Passive Mode (|||40889|)
---- Connecting data socket to (2001:9e8:202d:bf6f:9a9b:cbff:fe6f:27f5) port 40889
---> LIST
<--- 150 Opening ASCII mode data connection for '/bin/ls -lgA'.
WARNING: Certificate verification: The certificate is NOT trusted. The certificate issuer is unknown. (BF:D0:CE:F9:08:7F:9A:2F:66:F1:EF:60:AA:1C:FB:C5:A2:8D:7D:4C)
drwxrwxrwx 3 ftp ftp 232 Jan 6 20:15 keograms
drwxrwxrwx 3 ftp ftp 232 Jan 6 20:15 startrails
drwxrwxrwx 3 ftp ftp 232 Jan 6 20:15 videos
-rw-rw-rw- 1 ftp ftp 2801664 Jan 6 21:19 SaveImage-server-11388
-rw-rw-rw- 1 ftp ftp 2810028 Jan 6 21:23 SaveImage-server-11403
-rw-rw-rw- 1 ftp ftp 2812689 Jan 6 21:26 SaveImage-server-26077
-rw-rw-rw- 1 ftp ftp 2785280 Jan 6 21:55 SaveImage-server-27787
-rw-rw-rw- 1 ftp ftp 2801664 Jan 6 21:32 SaveImage-server-3047
-rw-rw-rw- 1 ftp ftp 2801664 Jan 6 21:30 SaveImage-server-30774
-rw-rw-rw- 1 ftp ftp 2801664 Jan 6 22:06 SaveImage-server-9586
-rw-rw-rw- 1 ftp ftp 2801664 Jan 6 21:59 SaveImage-server-96
-rw-rw-rw- 1 ftp ftp 2810816 Jan 6 22:09 image.jpg
<--- 226 Transfer complete.
-rw-rw-rw- 1 ftp ftp 27 Jan 6 22:01 testUpload.sh.txt
---> TYPE I
<--- 200 Type set to I.
---> EPSV
<--- 229 Entering Extended Passive Mode (|||40869|)
---- Connecting data socket to (2001:9e8:202d:bf6f:9a9b:cbff:fe6f:27f5) port 40869
---> STOR testUpload.sh-29035
<--- 150 Opening BINARY mode data connection for 'testUpload.sh-29035'.
WARNING: Certificate verification: The certificate is NOT trusted. The certificate issuer is unknown. (BF:D0:CE:F9:08:7F:9A:2F:66:F1:EF:60:AA:1C:FB:C5:A2:8D:7D:4C)
**** Peer closed connection
---- Connecting to ehcj2i7etqqinlyu.myfritz.net (2001:9e8:202d:bf6f:9a9b:cbff:fe6f:27f5) port 42817
<--- 220 FTP server ready.
---> FEAT
<--- 211- Extensions supported:
<--- UTF8
<--- MDTM
<--- SIZE
<--- AUTH TLS
<--- PBSZ
<--- PROT
<--- 211 end
---> AUTH TLS
<--- 234 Authentication method accepted
---> OPTS UTF8 ON
WARNING: Certificate verification: The certificate is NOT trusted. The certificate issuer is unknown. (BF:D0:CE:F9:08:7F:9A:2F:66:F1:EF:60:AA:1C:FB:C5:A2:8D:7D:4C)
<--- 530 Please login with USER and PASS.
---> USER radinkel
<--- 331 Password required for radinkel.
---> PASS XXXX
<--- 230 User radinkel logged in.
---> OPTS UTF8 ON
<--- 200 UTF8 ON
---> PBSZ 0
<--- 200 ok
---> PROT P
<--- 200 Data channel will be secured
---> CWD Spica
<--- 250 CWD command successful.
---> CWD allsky
<--- 250 CWD command successful.
---> TYPE I
<--- 200 Type set to I.
---> SIZE testUpload.sh-29035
<--- 213 0
---> EPSV
<--- 229 Entering Extended Passive Mode (|||44353|)
---- Connecting data socket to (2001:9e8:202d:bf6f:9a9b:cbff:fe6f:27f5) port 44353
---> STOR testUpload.sh-29035
<--- 150 Opening BINARY mode data connection for 'testUpload.sh-29035'.
WARNING: Certificate verification: The certificate is NOT trusted. The certificate issuer is unknown. (BF:D0:CE:F9:08:7F:9A:2F:66:F1:EF:60:AA:1C:FB:C5:A2:8D:7D:4C)
**** Peer closed connection
---- Connecting to ehcj2i7etqqinlyu.myfritz.net (2001:9e8:202d:bf6f:9a9b:cbff:fe6f:27f5) port 42817
<--- 220 FTP server ready.
---> FEAT
<--- 211- Extensions supported:
<--- UTF8
<--- MDTM
<--- SIZE
<--- AUTH TLS
<--- PBSZ
<--- PROT
<--- 211 end
---> AUTH TLS
<--- 234 Authentication method accepted
---> OPTS UTF8 ON
WARNING: Certificate verification: The certificate is NOT trusted. The certificate issuer is unknown. (BF:D0:CE:F9:08:7F:9A:2F:66:F1:EF:60:AA:1C:FB:C5:A2:8D:7D:4C)
<--- 530 Please login with USER and PASS.
---> USER radinkel
<--- 331 Password required for radinkel.
---> PASS XXXX
<--- 230 User radinkel logged in.
---> OPTS UTF8 ON
<--- 200 UTF8 ON
---> PBSZ 0
<--- 200 ok
---> PROT P
<--- 200 Data channel will be secured
---> CWD Spica
<--- 250 CWD command successful.
---> CWD allsky
<--- 250 CWD command successful.
---> TYPE I
<--- 200 Type set to I.
---> SIZE testUpload.sh-29035
<--- 213 0
---> EPSV
<--- 229 Entering Extended Passive Mode (|||35883|)
---- Connecting data socket to (2001:9e8:202d:bf6f:9a9b:cbff:fe6f:27f5) port 35883
---> STOR testUpload.sh-29035
<--- 150 Opening BINARY mode data connection for 'testUpload.sh-29035'.
WARNING: Certificate verification: The certificate is NOT trusted. The certificate issuer is unknown. (BF:D0:CE:F9:08:7F:9A:2F:66:F1:EF:60:AA:1C:FB:C5:A2:8D:7D:4C)
<--- 226 Transfer complete.
---> SITE UTIME 20250106210945 testUpload.sh-29035
**** gnutls_record_recv: Error in the pull function.
---- Connecting to ehcj2i7etqqinlyu.myfritz.net (2001:9e8:202d:bf6f:9a9b:cbff:fe6f:27f5) port 42817
<--- 220 FTP server ready.
---> FEAT
<--- 211- Extensions supported:
<--- UTF8
<--- MDTM
<--- SIZE
<--- AUTH TLS
<--- PBSZ
<--- PROT
<--- 211 end
---> AUTH TLS
<--- 234 Authentication method accepted
---> OPTS UTF8 ON
WARNING: Certificate verification: The certificate is NOT trusted. The certificate issuer is unknown. (BF:D0:CE:F9:08:7F:9A:2F:66:F1:EF:60:AA:1C:FB:C5:A2:8D:7D:4C)
<--- 530 Please login with USER and PASS.
---> USER radinkel
<--- 331 Password required for radinkel.
---> PASS XXXX
<--- 230 User radinkel logged in.
---> OPTS UTF8 ON
<--- 200 UTF8 ON
---> PBSZ 0
<--- 200 ok
---> PROT P
<--- 200 Data channel will be secured
---> CWD Spica
<--- 250 CWD command successful.
---> CWD allsky
<--- 250 CWD command successful.
---> TYPE I
<--- 200 Type set to I.
---> SIZE testUpload.sh-29035
<--- 213 27
---> SITE UTIME 20250106210945 testUpload.sh-29035
<--- 500 'SITE UTIME 20250106210945 testUpload.sh-29035': command not understood.
---> SITE UTIME testUpload.sh-29035 20250106210945 20250106210945 20250106210945 UTC
<--- 500 'SITE UTIME testUpload.sh-29035 20250106210945 20250106210945 20250106210945 UTC': command not understood.
---> DELE testUpload.sh.txt
<--- 250 DELE command successful.
---> TYPE A
<--- 200 Type set to A.
---> EPSV
<--- 229 Entering Extended Passive Mode (|||51165|)
---- Connecting data socket to (2001:9e8:202d:bf6f:9a9b:cbff:fe6f:27f5) port 51165
---> LIST
<--- 150 Opening ASCII mode data connection for '/bin/ls -lgA'.
WARNING: Certificate verification: The certificate is NOT trusted. The certificate issuer is unknown. (BF:D0:CE:F9:08:7F:9A:2F:66:F1:EF:60:AA:1C:FB:C5:A2:8D:7D:4C)
<--- 226 Transfer complete.
---> TYPE I
<--- 200 Type set to I.
---> RNFR testUpload.sh-29035
<--- 350 File exists, ready for destination name
---> RNTO testUpload.sh.txt
<--- 250 RNTO command successful.
---> QUIT
<--- 221 Goodbye.

Please remove 'Spica/allsky//testUpload.sh.txt' on your server.

[Radinkel]

When I switch to ftps, I get this:
Executing:
/home/admin/allsky/scripts/upload.sh --debug --remote-server /tmp/testUpload.sh.txt Spica/allsky/ testUpload.sh.txt testUpload.sh
Test upload to Remote Server FAILED with RET=99.

=================== RAW OUTPUT:

    Commands used are in: /home/admin/allsky/tmp/lftp_cmds/testUpload.sh.txt
    Upload output from '/tmp/testUpload.sh.txt:
       *** upload.sh: ERROR, RET=99:
    FILE_TO_UPLOAD='/tmp/testUpload.sh.txt'
    REMOTE_HOST='ehcj2i7etqqinlyu.myfritz.net'
    DIRECTORY='Spica/allsky/'
    TEMP_NAME='testUpload.sh-17717'
    DESTINATION_NAME='testUpload.sh.txt'

    ---- Resolving host address...
    ---- 2 addresses found: 2001:9e8:202d:bf6f:9a9b:cbff:fe6f:27f5, 92.117.244.67
    ---- Connecting to ehcj2i7etqqinlyu.myfritz.net (2001:9e8:202d:bf6f:9a9b:cbff:fe6f:27f5) port 42817
    **** gnutls_handshake: An unexpected TLS packet was received.
    cd: Fatal error: gnutls_handshake: An unexpected TLS packet was received.

[EricClaeys]

@Radinkel, we are making progress. Your post 2 hours ago worked, I assume with Protocol of ftp. What is in the FTP Settings field in the WebUI?

Until a day age, no one reported the An unexpected TLS packet was received. message so our challenge is to determine what set ... command eliminates that message. In my posts earlier I mentioned what I found in a quick Google search. There are likely other fixes as well. Once we know I will update testUpload.sh so it checks for that message and tells users what to do.

FYI, this isn't an Allsky problem but instead a problem determining how to properly connect to your server. Can you ask the company that provides your service?

[Radinkel]

set ssl:verify-certificate no: because first I got an error related to not trusted certificate
As suggested earlier in this post I tried set ssl:check-hostname false and set ftp:ssl-auth TLS without success.
I get the fatal error only with ftps (second Log). Using ftp (fist Log) gets no error, but ultra slow. The uploadtest lasts about 1 Minute!
Provider is Telekom and the FTP(S)-Server is on a Fritzbox. Both realy big companies, that say not my problem. I use WinSCP with same login data, without problems.

[EricClaeys]

@Radinkel, forgot to ask...
Can you provide me a temporary ftp login on your server with a different username and password, and a different home directory? That way I can more easily test.

[Radinkel]

well, I have become modest and humble... until I throw the laptop in the corner!
I did as you said, grep SaveImage-27120 /var/log/allsky.log , and now? Sorry there is no var/log/allsky.log

[EricClaeys]

The log file exists; if it didn't, grep would have given an error that the file didn't exist.
The file is a large text file that is recreated weekly so it doesn't get too large. It puts the old file in /var/log/allsky.log.1.
Run ls -l /var/log to see all the logs.

Your second grep implies the file is a binary file, i.e., it has non-printable characters. Can you attach the file?

[Radinkel]

You fox!
Give me precise instructions right away ;)
allsky.log
otherwise it won't work out for me....
Now you know EVERYTHING:
allsky.log

[EricClaeys]

There are a lot of these messages in the log file: No space left on device

You have run out of space on the server. This explains some of the errors.

[Radinkel]

Maybe because of tonns of files. I delete them regulary. Now there are 260MB free from 355 MB. Remember I want to keep only the last image, so 355 MB is enough. I dont use this NAS for something else.

[EricClaeys]

Have either of you tried FileZilla or another FTP client from your PC to the server to see if it works?

[Radinkel]

Does allsky check whether the file has "arrived"? That was the problem with the other software I told you about. It uploads and checks, but unfortunately a command from the Fritzbox is apparently ignored, and sends wrony reply "no such file", although it is there. see this post:
https://winscp.net/forum/viewtopic.php?t=33994

[EricClaeys]

Indirectly. Allsky uses the return code from the lftp commands to determine if they work. If put returns success we assume the file is there.
It appears that the put can say it failed but the saveImage- file actually was uploaded. I may add a check upon failure to see if the file is actually there.

If interested, take a look at ~/allsky/scripts/upload.sh for the commands.

[Radinkel]

It seems for me that only MDTM command does not work and use list instead. Nothing about put.

[Radinkel]

MDTM is not usesd in the upload.sh. It's 2:00 am, have to go to bed, it's your turn ;)

[jlfutari]

@EricClaeys - Yep I can connect to my Server with Filezilla without any issues