added support for secure TLS connections (#699)

* added support for secure TLS connections
on ESP32 and ESP8266 close #101

* added TLS test builds

* added documentation for TLS
fixed some some issues in the documentation

* added documentation for error code=54
This commit is contained in:
Leon Kiefer
2020-08-02 18:29:28 +02:00
committed by GitHub
parent 60d173ff51
commit b5527d2e7b
14 changed files with 180 additions and 40 deletions

1
.gitignore vendored
View File

@@ -1,2 +1,3 @@
.pio
.vscode
*_env.ini

View File

@@ -63,6 +63,7 @@ module.exports = {
'upload/pio',
'upload/arduino-ide',
'upload/portal',
'upload/advanced-configuration',
'upload/troubleshoot'
]
},

View File

@@ -1,6 +1,6 @@
# Integrate Home Assistant
## Auto discovery
So as to enable HASS auto discovery with MQTT you have to uncomment [ZmqttDiscovery](https://github.com/1technophile/OpenMQTTGateway/blob/0180a0dbd55ed8e0799e30ee84f68070a6f478fa/User_config.h#L99) in user_config.h
So as to enable HASS auto discovery with MQTT you have to uncomment [ZmqttDiscovery](https://github.com/1technophile/OpenMQTTGateway/blob/0180a0dbd55ed8e0799e30ee84f68070a6f478fa/User_config.h#L99) in User_config.h
And enable discovery on your MQTT integration definition in HASS.
OMG will use the auto discovery functionality of home assistant to create sensors and gateways into your HASS instance automaticaly.

View File

@@ -0,0 +1,34 @@
# Advanced configuration of the gateway
## Secure connection to the broker
By default the connection between the gateway and the MQTT broker is not encrypted and the identity of the broker is not verified.
For connections in a private local area network this might not be a big issue, but if you connect to a broker on the internet or the gateway is connected to a public network there is a high security risk.
To secure the connection, Transport Layer Security (TLS) can be used which is supported by most MQTT brokers.
Setting up and configuring TLS in the MQTT broker is a complex process and perhaps include creating a self-signed certificate.
The configuration of the broker is not covered here, you should look into the documentation of your broker.
### Prerequisites
The MQTT broker is configured for TLS and you have access to the CA certificate which was used to sign the MQTT broker certificate.
You are using ESP8266 or ESP32, for other boards TLS is not supported.
### Configure secure connection in the gateway
To enable the secure connection and use TLS uncomment `//#define SECURE_CONNECTION` in `User_config.h`.
Set `MQTT_SERVER` to the Common Name (CN) of the certificate of the broker.
This can be the hostname or the ip of the broker.
The CA certificate should be in PEM ascii format.
If your CA certificate is not in the correct format or you don't know the format, use `openssl` to convert the certificate to the correct format.
In `User_config.h` replace the `...` with the content of your certificate which is between the `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----` lines:
```cpp
const char* certificate CERT_ATTRIBUTE = R"EOF("
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
")EOF";
```
If you have no ntp server in your local network (included in the router) or not using dhcp, you should uncomment `//# define NTP_SERVER "pool.ntp.org"` to use a ntp server for time synchronization.
This is related to the `W: failed, ssl error code=54` error message, which indicate that the time of the esp is not correct.
You can know compile and upload to your board and the gateway should connect with TLS to your broker.

View File

@@ -8,10 +8,10 @@
* Unzip the libraries into your arduino libraries folder (example D:/Users/XXXX/Documents/Arduino/libraries)
* If necessary replace the spaces into each library folder by _: example rename “ESP32 BLE Arduino” folder to “ESP32_BLE_Arduino”
* Open the file main.ino from OpenMQTTGateway/main folder with the arduino IDE
* Change the settings and the desired gateways into user_config.h (uncomment the modules you want)
* Change the settings and the desired gateways into User_config.h (uncomment the modules you want)
*Example for the use of RF gateway*
```C++
```cpp
#define ZgatewayRF "RF" //ESP8266, Arduino, ESP32
//#define ZgatewayIR "IR" //ESP8266, Arduino, Sonoff RF Bridge
//#define ZgatewayLORA "LORA" //ESP8266, Arduino, ESP32
@@ -27,4 +27,4 @@
* Upload ➡️
* You should see the logs into the serial monitor
With an ESP if you did not set your network and mqtt parameters manualy you can now open the [web portal configuration](portal.md).
With an ESP if you did not set your network and mqtt parameters manually you can now open the [web portal configuration](portal.md).

View File

@@ -1,9 +1,8 @@
# Upload Binaries
This section is usefull if you want to directly flash your ESP from your desktop. Once flashed you can change wifi and broker settings.
This section is useful if you want to directly flash your ESP from your desktop. Once flashed you can change wifi and broker settings.
Nevertheless you will not be able to change advanced parameters, if you want to do so refer to [Upload from PlatformIO][pio] section.
Download the binary corresponding to your board and gateway [here](https://github.com/1technophile/OpenMQTTGateway/releases) from github and uncompress it.
Note that the -all binary packages are produced for tests only.
## ESP32
* Download the bootloader [here](https://github.com/espressif/arduino-esp32/raw/master/tools/sdk/bin/bootloader_dio_80m.bin)
@@ -13,7 +12,7 @@ https://www.espressif.com/en/products/hardware/esp32/resources
* Uncompress the package
* Execute `flash_download_tools`
* Choose ESP32 DownloadTool
* Set the files and the adress as below:
* Set the files and the address as below:
![Flash download tool](../img/OpenMQTTgateway_ESP32_binary_flash.png)
And set the parameters used by arduino IDE, we are able to upload to ESP32 a binary file containing OpenMQTTGateway.
* Set the config as above
@@ -33,6 +32,6 @@ https://github.com/marcelstoer/nodemcu-pyflasher/releases
* Click on *FlashNodeMCU*
The upload details appears.
With an ESP if you did not set your network and mqtt parameters manualy you can now open the [web portal configuration](portal.md).
With an ESP if you did not set your network and mqtt parameters manually you can now open the [web portal configuration](portal.md).
Note that to reset the wifi and mqtt settings you can check *yes, wipes all data*

View File

@@ -1,6 +1,6 @@
# Upload with PlatformIO
This section is usefull if you want to do advanced configuration of your project or if you choose an Arduino. Indeed the ESP family can be loaded directly without any configuration from your desktop.
Advanced configurations means changing the default used pins, the MQTT subjects and all the expert parameters that you can find in [user_config.h](https://github.com/1technophile/OpenMQTTGateway/blob/development/main/User_config.h) and in all [config_XX.h](https://github.com/1technophile/OpenMQTTGateway/tree/development/main).
Advanced configurations means changing the default used pins, the MQTT subjects and all the expert parameters that you can find in [User_config.h](https://github.com/1technophile/OpenMQTTGateway/blob/development/main/User_config.h) and in all [config_XX.h](https://github.com/1technophile/OpenMQTTGateway/tree/development/main).
If you don't have to change the default parameters except wifi and broker setting you can go directly to the [Load](load) section.
Download the [CODE](https://github.com/1technophile/OpenMQTTGateway/releases) from github.
@@ -28,9 +28,9 @@ default_envs = esp32dev-rf
If you don't know which env to activate you can refer to [devices](../prerequisites/devices).
The different listed configurations represents some standard environments, to overload them with special parameters or modules you can modify the config files. The definitions coming from [platformio.ini](https://github.com/1technophile/OpenMQTTGateway/blob/development/platformio.ini) file and config files are cumulative.
The main config file is [user_config.h](https://github.com/1technophile/OpenMQTTGateway/blob/development/main/User_config.h), added to it you have one config file per gateway, sensor or actuator, you will find them in the [main](https://github.com/1technophile/OpenMQTTGateway/tree/development/main) folder.
The main config file is [User_config.h](https://github.com/1technophile/OpenMQTTGateway/blob/development/main/User_config.h), added to it you have one config file per gateway, sensor or actuator, you will find them in the [main](https://github.com/1technophile/OpenMQTTGateway/tree/development/main) folder.
If you want to add more sensors or gateways to one `default_envs` you can uncomment them into [user_config.h](https://github.com/1technophile/OpenMQTTGateway/blob/d2dd6138558909b71cc44f69665340247bd5f356/main/User_config.h#L84) or add the modules directly into your environment definition of your .ini files.
If you want to add more sensors or gateways to one `default_envs` you can uncomment them into [User_config.h](https://github.com/1technophile/OpenMQTTGateway/blob/d2dd6138558909b71cc44f69665340247bd5f356/main/User_config.h#L84) or add the modules directly into your environment definition of your .ini files.
Example to add IR to `esp32dev-rf` add the `build_flags` below to the env definition:
``` ini
@@ -102,7 +102,7 @@ With the V0.9 we added the support of json for receiving and publishing.
Per default Json reception and Json publication is activated, the previous simple reception mode is also activated to avoid regression on commands.
You can deactivate Json or simple mode following theses instructions:
```C++
```cpp
#define jsonPublishing true //comment if you don't want to use Json publishing (one topic for all the parameters)
//example home/OpenMQTTGateway_ESP32_DEVKIT/BTtoMQTT/4XXXXXXXXXX4 {"rssi":-63,"servicedata":"fe0000000000000000000000000000000000000000"}
//#define simplePublishing true //comment if you don't want to use simple publishing (one topic for one parameter)
@@ -114,7 +114,7 @@ You can deactivate Json or simple mode following theses instructions:
```
If you are using platformio you can also comment the definitions above and define your parameters into platformio.ini file by setting the following `build_flags`:
```C++
```cpp
'-DjsonPublishing=true'
'-DjsonReceiving=true'
'-DsimpleReceiving=true'
@@ -138,4 +138,4 @@ With an ESP if you did not set your network and mqtt parameters manualy you can
::: warning Note
simpleReceiving on Arduino boards doesn't accept 64 bits MQTT values, you can only send 32bits values from the MQTT broker.
:::
:::

View File

@@ -62,3 +62,16 @@ This can be due to corruption of the ESP flash memory, try to erase flash and up
If you didn't find your answer here post a question to the forum:
[![Community forum](https://img.shields.io/badge/community-forum-brightgreen.svg)](https://community.openmqttgateway.com)
## ESP does not connect to broker with TLS enabled
If you get the following error:
`W: failed, ssl error code=54` ("Certificate is expired or not yet valid.")
This is most probable caused by the time of the esp is not correct/synchronized.
The esp uses the Network Time Protocol (NTP) to get the current time from a time server.
If you get this error ntp is not configured correctly in the gateway.
Uncomment `//# define NTP_SERVER "pool.ntp.org"` to set the `pool.ntp.org` as the time server.
You can also choose any other ntp time server you like.
It is normal that the time synchronization process takes some time and the MQTT connection will not be successful the first time.
If you set the ntp server for the gateway and keep getting the errors you should check your certificate validity duration.

View File

@@ -5,7 +5,7 @@ Subscribe to all the messages with mosquitto or open your MQTT client software:
` sudo mosquitto_sub -t +/# -v`
_NOTE: HM-10 or HM-11 module needed if you are not using ESP32; configure in `user_config.h`_
_NOTE: HM-10 or HM-11 module needed if you are not using ESP32; configure in `User_config.h`_
The BT gateway module for OpenMQTTGateway enables the detection of BLE beacons and their signal strength. Generally BLE devices will not broadcast if they are paired so you may need to ensure your beacons is unpaired before it will be seen by the gateway.

View File

@@ -10,12 +10,12 @@ Subscribe to all the messages with mosquitto or open your MQTT client software:
`mosquitto_sub -t +/# -v`
And press your IR remote control in front of the receiver led you should see the following messages for example:
```
```json
home/OpenMQTTGateway/IRtoMQTT {"value":875849879,"protocol":7,"protocol_name":SAMSUNG,"bits":32,"raw":"4534,4432,612,518,614,516,616,1618,618,1616,618,512,618,1618,608,524,612,518,616,514,618,512,616,1618,616,1618,618,514,616,1618,616,514,616,514,618,512,616,1618,618,1618,618,514,610,1622,616,514,618,514,614,516,616,1618,618,512,618,512,618,1616,550,580,618,1616,612,1624,618,1616,618"}
```
With an hexadecimal value:
```
```json
{"value":9938405643,"protocol":55,"bits":35,"hex":"0x25060090B","protocol_name":"TECO"}
```
@@ -50,9 +50,9 @@ If you dont want to use special parameters for IR just use value key, the pro
`mosquitto_pub -t home/OpenMQTTGateway/commands/MQTTtoIR -m '{"value":551489775}'`
NOTE: on arduino Uno most of the protocols are not enable per default due to memory constraints (it is not the case for MEGA), to enable them go to [user_config.h](https://github.com/1technophile/OpenMQTTGateway/blob/master/main/User_config.h) and uncomment the #define corresponding the protocols you want:
NOTE: on arduino Uno most of the protocols are not enable per default due to memory constraints (it is not the case for MEGA), to enable them go to [User_config.h](https://github.com/1technophile/OpenMQTTGateway/blob/master/main/User_config.h) and uncomment the #define corresponding the protocols you want:
```C++
```cpp
//#define IR_COOLIX
//#define IR_Whynter
//#define IR_LG
@@ -92,7 +92,7 @@ mosquitto_pub -t home/OpenMQTTGateway/commands/MQTTtoIR -m '{"value":551489775,"
## Send raw IR data by MQTT
2) If you use an arduino UNO enable `IR_Raw` by uncommenting the line 129 in user_config.h
2) If you use an arduino UNO enable `IR_Raw` by uncommenting the line 129 in User_config.h
`#define IR_Raw`
If you are using the uno you will have to comment other gateway like ZgatewayRF, ZgatewayBT and ZgatewayIR to keep enough memory
@@ -103,12 +103,16 @@ mosquitto_pub -t home/OpenMQTTGateway/commands/MQTTtoIR -m '{"raw":"8850,4450,60
With big raw array you may cross the limit of default payload size. In this case the gateway will not receive the message or will not send it to the broker.
In this case the best way is to use hex values instead, but if you can't you may change the parameters below:
In user_config.h replace:
`# define JSON_MSG_BUFFER 512`
`# define mqtt_max_packet_size 1024
In User_config.h replace:
```cpp
# define JSON_MSG_BUFFER 512
# define mqtt_max_packet_size 1024
```
by
`# define JSON_MSG_BUFFER 1280`
`# define mqtt_max_packet_size 1280`
```cpp
# define JSON_MSG_BUFFER 1280
# define mqtt_max_packet_size 1280
```
## Repeat the IR signal OpenMQTTGateway receive
So as to repeat the IR signal received by the gateway once set the following parameter to true in [config_IR.h](https://github.com/1technophile/OpenMQTTGateway/blob/091b317660fd201a30e2cd0e15424a13c5a6bd71/config_IR.h#L37)

View File

@@ -133,7 +133,7 @@ will make RF Bridge send a signal with the use of advanced parameters defined in
RF2 gateway enables to send command to RF devices with the KAKU protocol. DIO chacon devices are an example.
It uses the same pinout as the RF gateway and both gateways can be used on the same setup.
Receiving RF codes with the KAKU protocol is not compatible with ZgatewayRF , so as to get the code of your remotes you should comment ZgatewayRF in user_config.h.
Receiving RF codes with the KAKU protocol is not compatible with ZgatewayRF , so as to get the code of your remotes you should comment ZgatewayRF in User_config.h.
Transmitting can be done with both ZgatewayRF and ZgatewayRF2
### Receiving data from KAKU signal

View File

@@ -56,7 +56,7 @@
#define Base_Topic "home/"
/*-------------DEFINE YOUR NETWORK PARAMETERS BELOW----------------*/
/*-------------DEFINE YOUR NETWORK PARAMETERS BELOW----------------*/
//#define NetworkAdvancedSetup true //uncomment if you want to set advanced network parameters for arduino boards, not uncommented you can set the IP and mac only
#ifdef NetworkAdvancedSetup // for arduino boards advanced config
@@ -69,7 +69,7 @@ const byte subnet[] = {255, 255, 255, 0};
#endif
#if defined(ESP8266) || defined(ESP32) // for nodemcu, weemos and esp8266
//#define ESPWifiManualSetup true //uncomment you don't want to use wifimanager for your credential settings on ESP
//# define ESPWifiManualSetup true //uncomment you don't want to use wifimanager for your credential settings on ESP
#else // for arduino boards
const byte ip[] = {192, 168, 1, 99};
const byte mac[] = {0xDE, 0xED, 0xBA, 0xFE, 0x54, 0x95}; //W5100 ethernet shield mac adress
@@ -100,7 +100,7 @@ const byte mac[] = {0xDE, 0xED, 0xBA, 0xFE, 0x54, 0x95}; //W5100 ethernet shield
/*-------------DEFINE YOUR MQTT PARAMETERS BELOW----------------*/
//MQTT Parameters definition
#if defined(ESP8266) || defined(ESP32) || defined(__AVR_ATmega2560__) || defined(__AVR_ATmega1280__)
# define parameters_size 20
# define parameters_size 30
# define mqtt_topic_max_size 100
# define mqtt_max_packet_size 1024
#else
@@ -109,6 +109,16 @@ const byte mac[] = {0xDE, 0xED, 0xBA, 0xFE, 0x54, 0x95}; //W5100 ethernet shield
# define mqtt_max_packet_size 128
#endif
// activate the use of TLS for secure connection to the MQTT broker
// MQTT_SERVER must be set to the Common Name (CN) of the broker's certificate
//#define SECURE_CONNECTION
#ifdef SECURE_CONNECTION
# define MQTT_DEFAULT_PORT "8883"
#else
# define MQTT_DEFAULT_PORT "1883"
#endif
#ifndef MQTT_USER
# define MQTT_USER "your_username"
#endif
@@ -119,7 +129,30 @@ const byte mac[] = {0xDE, 0xED, 0xBA, 0xFE, 0x54, 0x95}; //W5100 ethernet shield
# define MQTT_SERVER "192.168.1.17"
#endif
#ifndef MQTT_PORT
# define MQTT_PORT "1883"
# define MQTT_PORT MQTT_DEFAULT_PORT
#endif
#ifdef SECURE_CONNECTION
# if defined(ESP8266) || defined(ESP32)
# if defined(ESP32)
# define CERT_ATTRIBUTE
# elif defined(ESP8266)
# define CERT_ATTRIBUTE PROGMEM
# endif
// The root ca certificate used for validating the MQTT broker
// The certificate must be in PEM ascii format
const char* certificate CERT_ATTRIBUTE = R"EOF("
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
")EOF";
// specify a NTP server here or else the NTP server from DHCP is used
//# define NTP_SERVER "pool.ntp.org"
# else
# error "only ESP8266 and ESP32 support SECURE_CONNECTION with TLS"
# endif
#endif
#if defined(ESP8266) || defined(ESP32)

View File

@@ -147,12 +147,16 @@ int failure_number_mqtt = 0; // number of failure connecting to MQTT
#ifdef ESP32
# include <ArduinoOTA.h>
# include <FS.h>
# include <WiFi.h>
# include <WiFiUdp.h>
# include <SPIFFS.h>
# include <esp_wifi.h>
# include "SPIFFS.h"
# include "esp_wifi.h"
# ifdef SECURE_CONNECTION
# include <WiFiClientSecure.h>
WiFiClientSecure eClient;
# else
# include <WiFi.h>
WiFiClient eClient;
# endif
# include <Preferences.h>
# include <WiFiManager.h>
Preferences preferences;
@@ -166,7 +170,12 @@ Preferences preferences;
# include <ESP8266WiFi.h>
# include <FS.h>
# include <WiFiManager.h>
# ifdef SECURE_CONNECTION
WiFiClientSecure eClient;
X509List caCert(certificate);
# else
WiFiClient eClient;
# endif
# ifdef MDNS_SD
# include <ESP8266mDNS.h>
# endif
@@ -459,6 +468,11 @@ void connectMQTT() {
failure_number_mqtt++; // we count the failure
Log.warning(F("failure_number_mqtt: %d" CR), failure_number_mqtt);
Log.warning(F("failed, rc=%d" CR), client.state());
#if defined(SECURE_CONNECTION) && defined(ESP32)
Log.warning(F("failed, ssl error code=%d" CR), eClient.lastError(nullptr, 0));
#elif defined(SECURE_CONNECTION) && defined(ESP8266)
Log.warning(F("failed, ssl error code=%d" CR), eClient.getLastSSLError());
#endif
delay(5000);
#if defined(ESP8266) || defined(ESP32)
disconnection_handling(failure_number_mqtt);
@@ -522,7 +536,9 @@ void setup() {
Log.trace(F("OpenMQTTGateway ip: %s" CR), WiFi.localIP().toString().c_str());
setOTA();
# ifdef SECURE_CONNECTION
setupTLS();
# endif
#else // In case of arduino platform
//Launch serial for debugging purposes
@@ -546,8 +562,8 @@ void setup() {
long port;
port = strtol(mqtt_port, NULL, 10);
Log.trace(F("Port: %l" CR), port);
client.setServer(mqtt_server, port);
Log.trace(F("Mqtt server: %s" CR), mqtt_server);
client.setServer(mqtt_server, port);
#endif
setup_parameters();
@@ -733,10 +749,10 @@ void setOTA() {
# endif
});
ArduinoOTA.onEnd([]() {
Log.trace(F("\nOTA done" CR));
# if defined(ZgatewayBT) && defined(ESP32)
startProcessing();
# endif
Log.trace(F("\nOTA done" CR));
# if defined(ZboardM5STICKC) || defined(ZboardM5STACK)
M5Display("OTA done", "", "");
# endif
@@ -762,6 +778,20 @@ void setOTA() {
});
ArduinoOTA.begin();
}
# ifdef SECURE_CONNECTION
void setupTLS() {
# if defined(NTP_SERVER)
configTime(0, 0, NTP_SERVER);
# endif
# if defined(ESP32)
eClient.setCACert(certificate);
# elif defined(ESP8266)
eClient.setTrustAnchors(&caCert);
eClient.setBufferSizes(512, 512);
# endif
}
# endif
#endif
#if defined(ESPWifiManualSetup)

View File

@@ -20,7 +20,7 @@ extra_configs =
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;Uncomment the env line corresponding to your board and modules required, ;
;you can also adapt the modules by removing the corresponding lines in the env detail ;
; if you go to the build flag section of your env you will see that some user_config.h ;
; if you go to the build flag section of your env you will see that some User_config.h ;
; parameters can be overwritten here, for example the gateway name. ;
; If you want to avoid the lost of your environments at each update you can put them ;
; into a separate file called prod_env.ini, it will be automatically read by pio ;
@@ -57,7 +57,9 @@ extra_configs =
;default_envs = uno-rf
;default_envs = uno-fastled
;default_envs = atmega-all
;default_envs = test-manual-wifi
;default_envs = manual-wifi-test
;default_envs = esp32dev-tls-test
;default_envs = nodemcuv2-tls-test
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; ENVIRONMENTS PARAMETERS ;
@@ -515,7 +517,7 @@ build_flags =
'-DGateway_Name="OpenMQTTGateway_ESP8266_RF-CC1101"'
board_build.flash_mode = dout
[env:test-manual-wifi]
[env:manual-wifi-test]
platform = ${com.esp8266_platform}
board = nodemcuv2
lib_deps =
@@ -529,6 +531,29 @@ build_flags =
'-DGateway_Name="OpenMQTTGateway_TEST_MANUAL_WIFI"'
board_build.flash_mode = dout
[env:esp32dev-tls-test]
platform = ${com.esp32_platform}
board = esp32dev
lib_deps =
${com-esp.lib_deps}
build_flags =
${com-esp.build_flags}
'-DSECURE_CONNECTION'
'-DGateway_Name="OpenMQTTGateway_TEST_TLS"'
board_build.flash_mode = dout
[env:nodemcuv2-tls-test]
platform = ${com.esp8266_platform}
board = nodemcuv2
lib_deps =
${com-esp.lib_deps}
build_flags =
${com-esp.build_flags}
'-DSECURE_CONNECTION'
'-DGateway_Name="OpenMQTTGateway_TEST_TLS"'
board_build.flash_mode = dout
[env:rf-wifi-gateway]
platform = ${com.esp8266_platform}
board = nodemcuv2