SSLCertificateManagement description and explanation is needed here.

[Hugocourr]

Hello all.
First of all this is my first time posting on forums in github so if my way of posting is not as it should be, don't hesitate to point it out, i'll take it into account for my next posts.
So, i'm having an issue connecting to a website using MKRGSM 1400 with the GSMClient::connectSSL function. I tried to connect on other website such as this one: "https://webhook.site/" and it worked perfectly, allowing me to send various GET and POST http request.
The thing is, to post or request anything on this website i need to put an authentification, i know the auth, and as i understand it has to go in the "POST" http request. Here is a screen of the website when i try to access it using chrome:

The problem being that, of course, i can't send a POST request without being connected to the domain wich i can't currently do.
I noticed that the certificate is quite particuliar for this website:

So i'm trying to upload on my board this sketch that is around here:
https://github.com/arduino-libraries/MKRGSM/tree/master/examples/SSLCertificateManagement
but i don't understand where i shoud search and what i am supposed to get or what it exactly is for that matter. Can anybody explain this to me? what should i use to find the proper files?
Just in case, my code is attached to the post (the certificateManagement functions are not used on it for now as i don't understand them and don't want to permanently ruin the certificates registered on my board's hardware):

#include <MKRGSM.h> // GSM library
#include <Wire.h>
#include <FlashStorage.h>
#include <ArduinoJson.h>
#include "arduino_secrets.h"

//Donnée pour connexion au server
//char server[] = "webhook.site";
char server[] = "push.som2m.com";

//char path[] = "";
char path[] = "...............";//i just replaced it, i don't know if it was a wise choice (or useful) to show that
char authorization[] = "Authorization: i don't think i should show that";

GPRS gprs;
GSM gsmAccess;

GSMSSLClient client (true);

void setup() {
  Serial.begin(9600);

  while (!Serial) {};

  //Enter Your SIM Card details below
  char apn[] = "";
  char login[] = "";
  char pwd[] = "";

  bool Connected = false;
  while (!Connected) {
    if ((gsmAccess.begin() == GSM_READY) & (gprs.attachGPRS(apn, login, pwd) == GPRS_READY)) {
      Connected = true;
      Serial.println(F("connected!"));
    }
    else
    {
      Serial.println(F("not connected :("));
      delay(1000);
    }
  }

  Serial.println(F("Connection..."));
  // Connect to HTTP server
  int err = client.connectSSL(server, 443);
  while (err == 0) {
    Serial.println("Error, retrying:"); //Code always stuck here: on domain connection.
    delay(10000);
    err = client.connectSSL(server, 443);
  }

  while (client.available()) {
    char c = client.read();
    Serial.print(c);
  }
  Serial.println("Connected!");

  StaticJsonDocument<384> doc;
  /*
  JSON created for testing purposes here, no need to show it.
  */


  client.print("POST ");
  client.print(path);
  client.println(" HTTP/1.1");
  //client.println("User-Agent: mw.doc.bulk-update (Arduino MKR GSM 1400)");//is this line useful?
  client.print("Host: ");
  client.println(server);
  client.println("Connection: keep-alive");
  client.println("Content-Type: application/json");
  client.println(authorization);
  client.print("Content-Length: ");
  client.println(measureJson(doc));
  client.println();
  // Write JSON document
  serializeJson(doc, client);
}

void loop() {
    if (!client.available() && !client.connected()) {

    Serial.println();

    Serial.println("disconnecting.");

    client.stop();
    while (1);
  }
}

[Hugocourr]

Hi all!
I figured some things out about certificate management on the MKRGSM 1400 and finally managed to connect my board to the domain. I thought I'll document the process here. Once again, I'll try to format my code and all in order to make it easy to read but I'm still a newbie around here so don't expect too much :).

So! this example (aka SSLCertificateManagement) is supposed to set as "trusted" self signed certificate on your board. If you are like me one week ago this could be chinese to you, so a little explanation of what a SSL connexion is, is required. Basically what happen when you connect via SSL is (copy pasted from IBM website):

The client sends a request to the server for a secure session. The server responds by sending its X.509 digital certificate to the client. (wich is given to him by an authority that delivers those)
The client receives the server's X.509 digital certificate.
The client authenticates the server, using a list of known certificate authorities.
The client generates a random symmetric key and encrypts it using server's public key.
The client and server now both know the symmetric key and can use the SSL encryption process to encrypt and decrypt the information contained in the client request and the server response.

What is described above is a classic SSL connection meaning that THIS IS NOT A SELF SIGNED CERTIFICATE as it has trusted roots (more on that later).
When using a self signed certificate, YOU create the certificate and YOU say to the board "I know it doesn't have any root certificates of confidence, but i know this certificate and you can trust it".

Just to be clear: i won't describe how to work with self signed certificates as i simply don't know how it works, if you keep reading this tutorial, it means that the connection to the domain you want to connect to, has trusted root certificate.

But, how do i know if the domain i want to connect to has root certificates? Well, like that: in chrome/firefox/any (or most?) navigator when you connect to most website nowadays this is what you see in the top of your screen in the bar:

Notice the lock? this means we are using Https connection. If you click on it something similar to this should appear:

Now what you need is to display the certificate (even registering it on your computer). I won't explain how to do it using all navigator, as you can find it pretty easily on internet and it's not the hard part.

What we want for our board, is the root certificate, so we are going to download it in order to visualise it :

What is shown by the arrow is the ROOT certificate for the certificate of github, ie, the one that says: i delivered this certificate to github, i know they are who they say they are, you can connect to them. I take this one as it'll allow to go to more website than just the github one.

note: i know this certificate is already on the board, but we'll see how to erase it and add it again.

Download the certificate alone in PEM format, it should look like this:

Congrats! you've done the first part of finding the certificate ,now we need to format it in order to input it to the MKR.

For that i used this website: https://holtstrom.com/michael/tools/hextopem.php and imputed the certificate in the right window:
Why this website in particular? Because PEM is 64 based encoded so a simple ASCII to HEX translator won't work

PLEASE NOTE: DO NOT INCLUDE THE -----BEGIN CERTIFICATE----- AND -----END CERTIFICATE----- PARTS IN THE CONVERSION

It should output something like that when clicking on "PEM TO HEX":

Copy and paste the output in a text file, we still need to format it a bit more.

Now, we need to format it in order to declare it as a const uint8_t array in th arduino code. For that i simply used this program on spyder, a python IDE:

f=open("path_to_non_formated_certif.txt", "r");
array=f.read();
cntarray=0;
Tablo="" ;
while cntarray<=len(array)-4:
    Tablo += '0';
    Tablo += 'x';
    Tablo += array[cntarray:cntarray+2];
    Tablo += ',';
    Tablo += ' ';
    cntarray=cntarray+2;
    if (cntarray%50)==0:
        Tablo += '\n';

Tablo += '0';
Tablo += 'x';
Tablo += array[cntarray:cntarray+2];

f = open("path_to_formated_certif.txt", "w");
f.write(Tablo);
f = open("path_to_formated_certif.txt", "a");
f.write("\n");
f.write("size of the certificate: ");
f.write(f"{len(array)/2}");
f.close();

Now, a file at path_to_formated_certif.txt was created, the file looks like this (except for the size of the certificate that i wrote in french at first)

Note: this ISN'T the certificate we saw earlier, it's another one i formatted previously so don't worry if the bytes don't match

Congrats! Part two is done, you made a formatted certificate that the MKR will take as input for Certificate!

Now, you may ask How do i implement it in the board Hardware? for that too, here is a custom sketch that i wrote:

//This code is supposed to manage root certificates on th SARA-U2 module of he MKRGSM
#include <MKRGSM.h> // GSM library

char server[] = "..................."; //you should know what goes there
char path[] = "......................."; //you should know what goes there

GPRS gprs;
GSM gsmAccess (true);
GSMSSLClient client ;

unsigned long getTime() {
  return gsmAccess.getTime();
}

void setup() {

  Serial.begin(9600);

  const uint8_t Certificat_racine[] { // it means root certificate in French ;)
    
  };

  while (!Serial) {};

  bool Connected = false;
  while (!Connected) {
    if ((gsmAccess.begin() == GSM_READY) & (gprs.attachGPRS("", "", "") == GPRS_READY)) {
      Connected = true;
      Serial.println(F("Connectée!"));
    }
    else
    {
      Serial.println(F("Pas connnectée :("));
      delay(1000);
    }
  }
  
  
  //In order to test if connection succeed once the certificate is updated
  /*
  Serial.println(F("Connecting"));
  // Connect to HTTPs server
  int err = client.connectSSL(server, 443);
  while (err == 0) {
    Serial.println("no working, sad times");
    delay(10000);
    err = client.connectSSL(server, 443);
  }
  Serial.println("Connection success! Party time!.");
  while (1);*/


  MODEM.sendf("AT+USECMNG=3");
  MODEM.waitForResponse(1000);



  /*
    Comment what's below in order to just read the board registered root certificate
*/
//This delete the certificate labeled as ROOT_CERTIFIATE_NAME in the board. Comment it if you don't want to delete any certificate.
  MODEM.sendf("AT+USECMNG=2,0,\"ROOT_CERTIFIATE_NAME\"");  
  MODEM.waitForResponse(1000);

// adds the certificate ROOT_CERTIFIATE_NAME to the Sara-U2 module registered root certificates.
  MODEM.sendf("AT+CMEE=2");
  MODEM.waitForResponse(100);
  MODEM.sendf("AT+USECMNG=0,0,\"ROOT_CERTIFIATE_NAME\",ROOT_CERTIFIATE_NAME_SIZE");
  MODEM.waitForResponse(100);
  MODEM.write(Certificat_racine, (size_t)ROOT_CERTIFIATE_NAME_SIZE);
  MODEM.waitForResponse(100);


  while (1);

}

void loop() {
  // put your main code here, to run repeatedly:

}

You see that big text file with all those bytes (path_to_formated_certif.txt)? Copy it in the certificat_racine and update the size (ROOT_CERTIFIATE_NAME_SIZE) of the certificate where needed.
Also, i don't know if that's important, but just to be sure, make sure the name (ROOT_CERTIFIATE_NAME) is the one of the certificate that you're uploading, with space replaced by "_", because i don't think the SARA_U2 module likes spaces and i don't wanna try.
For those eager to play a bit and understand the module, search for this: ublox20AT20Commands20Manual.989903893.pdf (little int: for what we are doing go to section 26 page 621).
I think the arduino code is self explanatory and i won't go in detail about it. At least not now. Maybe latter if needed.

I just want to point out that i am a total newbie in this domain, and there might be some errors in my code and you can surely optimise a lot of stuff here and there. But for me, it worked like a charm and i can now post on classic https website.
I'm open to any question and remarks you guys might have :) .

Thank you for reading and have fun!

[kekko7072]

Thanks, I think I am facing the same issue with firebase functions, I don't understand why webhook.site works also with https request. Can you explain me?