How to switch to SSL in an open TCP connection

[FalconMuc]

Answers checklist.

• I have read the documentation ESP-AT Programming Guide and the issue is not addressed there.
• I have used the latest released firmware or have updated my ESP-AT branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

Hi everybody,
in several situations it is necessary to switch from un unencrypted connection “TCP” to an encrypted connection “SSL”. Two examples for this are the communication with a SMTP server (command STARTTLS) or with a FTP server (-> FTPS command AUTH TLS).

I already tested direct SSL connections e.g. with “implicit” SSL connections successfully. This is possible if the server provides a dedicated port to directly connect via SSL to (e.g. implicit SSL connection to a SMTP server, port 465). This works really good e.g for sending encrypted emails.

In several situations the server does not offer such a dedication port for SSL for implicit SSL, but offers “explicit” SSL (SMTP or FTP(S)). in several documents “implicit” SSL is called the “old” way. In this situation of “explicit SSL” the connection is started without encryption and during the communication with the server a special command e.g. “STARTTLS (SMTP)” or “AUTH TLS (FTPS)” is used to switch to an encrypted connection.

Now to the ESP-AT. “Implicit” SSL works good so far, here the encrypted connection is established from the very beginning. But how can I switch to an encrypted connection while a normal „TCP“ connection is „running“. I must not disconnect from the server first to start a “SSL” connection because I have to initiate the encrypted connection via a command.

Your answers are highly appreciated.
Thomas

[ustccw]

@FalconMuc have you tried the multiple connections by AT+CIPMUX=1. You can establish multiple connections by AT+CIPSTART command under multiple connection mode, refer to Examples for more details.

[FalconMuc]

Hi ustccw,

thank you for your thoughts, they are highly appreciated!

I haven't tried that yet, but I’m a bit skeptical if this could work. I already worked with AT+CIPMUX=1 in other applications like FTP where two simultaneous connections are necessary.

The SMTP server receives e.g. the command STARTTLS and waits for the negotiation of the encryption key on the same socket. After the successful negotiation the encrypted communication continues on the same socket.

So I’m afraid trying to start a second “encrypted” SSL connection after sending the STARTTLS will not work, because the server will open a brand new socket that always starts unencrypted. The ESP-AT will probably be unable to start the SSL connection.

Thank you again for your thoughts
Thomas

[ustccw]

no worries. If that cannot work, you also can run the smtp client by adding a self-defined AT command, refer to smtp client and How to add user-defined AT commands for more details.

[FalconMuc]

Hi ustccw,

thank you again for your creative help.

The situation is, that I have already implemented a complete SMTP client as well as a complete FTP client on the basis of the ESP-AT command library. Both clients work well and reliable with unencrypted connections. If there would be an AT command or another solution to do a transition from a "TCP" (unencrypted) connection to a "SSL" (encrypted) connection, it would basically be just one more command to implement the complete encrypted functionality.

So implementing a new AT command and using the smtp client solution you suggested would only kind of solve half of the problem (the FTP client would still have not SSL/TLS encryption). On the other hand the smtp client you suggested implements a second complete SSL/TLS functionality with the mbetTLS library it uses. The ESP-AT library already includes such a SSL/TLS functionality for the "SSL" connections, right?

I find your idea of implementing a new AT command interesting. But it would only be necessary to be able to switch from "TCP" to "SSL" during a running "TCP" connection. For this I have no information how the ESP-AT handles such transitions of unencrypted to encrypted sockets and I'm not sure if such a AT command incooperating with the existing AT commands is possible.

Thanks again for your help.
Thomas

[ustccw]

Hello @FalconMuc ,
From a code perspective, it is possible to switch from a TCP connection to an SSL connection, theoretically making it compatible with other AT commands.
You can roughly obtain the tcp socket fd by calling the hidden API: int32_t esp_at_get_socket_by_link_id(uint8_t id);, and then further establish the TLS connection by calling mbedtls-related APIs (similar to our esp-tls implement).

However, this is a highly challenging task. You need to be very familiar with the mbedtls process, resource management (especially on resources free), and the protocol switching process of SMTP/FTP. Therefore, it is generally not recommended to use it directly in AT commands.

Alternatively, you can consider running the SMTP/FTP protocol stack in MCU side.

[FalconMuc]

Hi @ustccw,

thank you again for your help.

We have already implemented the complete SMTP stack and FTP stack on the host MCU. The SMTP stack and FTP stack is running on the host MCU. It is working without problems.

The host MCU is communicating with the ESP-C2 (ESP8684) using the ESP-AT and it is working great!

But it only works WITHOUT encryption (SSL/TLS). For encrypted SMTP and/or FTP it is necessary to switch to SSL/TLS during a running connection.

To develop a complete new AT command with e.g. your "smtp_client example" to send an encrypted email doesn’t really make sense, because we already implemented everything for the SMTP/FTP stack on the host MCU.

Some important additional information: Some SMTP servers offer direct (implicit) SSL/TLS connection via port 465. So when I connect to port 465 via a "AT+CIPSTART="SSL",..." I can already send SSL/TLS encrypted emails. But this approach is NOT RECOMMENDED. It is recommended to use the SMTP command "STARTTLS" to start the encryption. With FTP it is the same.
So the ESP-AT already includes everything to setup a working SSL/TLS connection to a SMTP server. It is only a question of "WHEN" the SSL/TLS negotiation starts. With the "AT+CIPSTART="SSL",..." command the SSL/TLS negotiation starts immediately after the standard "TCP" connection is established. The only thing necessary would be to NOT DO the negotiation immediately but at a later point in time (e.g. after sending the "STARTTLS" command for the SMTP server or after the "AUTH TLS" command for the FTP server).

For our project no certificate is necessary but only the encryption.

Thank you for your help.
Thomas

[ustccw]

OK, you can try that highly challenging task i mentioned above. or run mbedTLS protocol stack on MCU.