Merge branch 'development' into master

README.md

@@ -33,13 +33,17 @@ These scripts were tested in
 * a Raspberry Pi 3B / 3B+ / 4B with Raspbian Stretch and its internal Wifi chip
 * a Raspberry Pi 3B / 3B+ Raspberry Pi OS Buster (previously called Raspbian) and its internal Wifi chip
 * a Raspberry Pi 3B+ + USB-WIFI with an image from [here](https://www.offensive-security.com/kali-linux-arm-images/)
-* Ubuntu 18.04.3 64Bit in VirtualBox on Win10 with a [cheap RTL8188CU Wifi Adapter](http://s.click.aliexpress.com/e/KrKIoPdI) connected to the VM
+* Ubuntu 18.04.3 64Bit in VirtualBox on Win10 with a cheap RTL8188CU Wifi Adapter connected to the VM
 
 Any Linux with a Wifi adapter which can act as an Access Point should also work. Please note that we have tested the Raspberry Pi with clean installations only. If you use your Raspberry Pi for anything else, we recommend using another SD card with a clean installation.
 
 ## PROCEDURE
 
-On January 28th, 2019, Tuya started [distributing a patch](https://www.heise.de/newsticker/meldung/Smart-Home-Hack-Tuya-veroeffentlicht-Sicherheitsupdate-4292028.html) that prevented older versions of tuya-convert from completing successfully. We have since developed a work around to enable OTA flashing once again, but there is always the possibility that Tuya will respond with yet another patch. To ensure the best chance of success, **do not connect your device with the official app** as it may automatically update the device, preventing you from flashing with tuya-convert. It is up to the individual brands to update their firmware, so some devices may be affected sooner than others.
+On January 28th, 2019, Tuya started [distributing a patch](https://www.heise.de/newsticker/meldung/Smart-Home-Hack-Tuya-veroeffentlicht-Sicherheitsupdate-4292028.html) that prevented older versions of tuya-convert from completing successfully. We have since developed a work around to enable OTA flashing on some newer devices, but Tuya has since released [yet another patch](https://github.com/ct-Open-Source/tuya-convert/issues/483).
+
+To ensure the best chance of success, **do not connect your device with the official app** as it may automatically update the device, preventing you from flashing with tuya-convert. It is up to the individual brands to update their firmware, so some devices may be affected sooner than others.
+
+Unfortunately many devices have already been shipping with the new patched firmware, evident by a PSK ID beginning with 02 in smarthack-psk.log. There is no workaround at this time. Additionally, manufacturers have been silently switching from the ESP82xx based modules to other chipsets, making it impossible to install alternative ESP firmware on these devices.
 
 ### INSTALLATION
     # git clone https://github.com/ct-Open-Source/tuya-convert
@@ -50,7 +54,7 @@ On January 28th, 2019, Tuya started [distributing a patch](https://www.heise.de/
 BE SURE THE FIRMWARE FITS YOUR DEVICE!
 1. Place your binary file in the `/files/` directory or use one of the included firmware images.
 
-	Currently a [Tasmota](https://github.com/arendst/Tasmota) `tasmota-wifiman.bin` build is included in the Tuya-Convert package. You can easily update to the [current maintenance release](http://thehackbox.org/tasmota) via OTA **after** the flashing process completes successfully. The included binary does not have any specific hardware configured. Once flashed using Tuya-Convert you will need to configure your device(s) properly. Please note that while we include this firmware for your convenience, we are not affiliated with the Tasmota project and cannot provide support for post installation issues. Please refer to the [Tasmota project](https://github.com/arendst/Tasmota) and [its documentation](http://tasmota.com) for configuration and support. **IMPORTANT:** If you still want to update the tasmota binary before using Tuya-Convert, always pickup `tasmota-wifiman.bin`. Never ever use `tasmota-minimal.bin` as you will brick your device.
+	Currently a [Tasmota](https://github.com/arendst/Tasmota) `tasmota-lite.bin` build is included in the Tuya-Convert package. You can easily update to the [current maintenance release](http://thehackbox.org/tasmota) via OTA **after** the flashing process completes successfully. The included binary does not have any specific hardware configured. Once flashed using Tuya-Convert you will need to configure your device(s) properly. Please note that while we include this firmware for your convenience, we are not affiliated with the Tasmota project and cannot provide support for post installation issues. Please refer to the [Tasmota project](https://github.com/arendst/Tasmota) and [its documentation](http://tasmota.com) for configuration and support. **IMPORTANT:** If you still want to update the tasmota binary before using Tuya-Convert, always pickup `tasmota-lite.bin`. **Never ever** use `tasmota-minimal.bin` as you will brick your device.
 
 	An ESPurna [1.13.5](https://github.com/xoseperez/espurna/releases/tag/1.13.5) binary is also included (`espurna-base.bin`). Like before, the binary included does not have any specific hardware defined. Once flashed using Tuya-Convert you can update to the device-specific version via any of the means that ESPurna provides (OTA, web interface update, update via telnet or MQTT). Please refer to the [ESPurna project page](http://espurna.io) for more info and support.
 

install_prereq.sh

@@ -3,12 +3,12 @@ set -euo pipefail
 
 debianInstall() {
 	sudo apt-get update
-	sudo apt-get install -y git iw dnsmasq hostapd screen curl build-essential python3-pip python3-setuptools python3-wheel python3-dev mosquitto haveged net-tools libssl-dev
+	sudo apt-get install -y git iw dnsmasq rfkill hostapd screen curl build-essential python3-pip python3-setuptools python3-wheel python3-dev mosquitto haveged net-tools libssl-dev
 	sudo python3 -m pip install --user --upgrade paho-mqtt tornado git+https://github.com/drbild/sslpsk.git pycryptodomex
 }
 
 archInstall() {
-	sudo pacman -S --needed git iw dnsmasq hostapd screen curl python-pip python-wheel python-pycryptodomex python-paho-mqtt python-tornado mosquitto haveged net-tools openssl
+	sudo pacman -S --needed git iw dnsmasq util-linux hostapd screen curl python-pip python-wheel python-pycryptodomex python-paho-mqtt python-tornado mosquitto haveged net-tools openssl
 	sudo python -m pip install --user --upgrade git+https://github.com/drbild/sslpsk.git
 }
 

scripts/dr_tuya.sh

@@ -0,0 +1,67 @@
+#!/bin/bash
+
+NON_ESP_WARNING="WARNING: it appears this device does not use an ESP82xx and therefore cannot install ESP based firmware"
+
+get_macs_from_wifi_log () {
+	grep AP-STA-CONNECTED smarthack-wifi.log |
+	cut -d ' ' -f3 |
+	sort | uniq
+}
+
+get_ouis_from_wifi_log () {
+	get_macs_from_wifi_log |
+	tr -d : | grep -Po "^.{6}"
+}
+
+psk_02 () {
+	grep -q "ID: 02" smarthack-psk.log
+}
+
+web_non_esp () {
+	grep -q "$NON_ESP_WARNING" smarthack-web.log
+}
+
+udp_non_esp () {
+	grep -q "$NON_ESP_WARNING" smarthack-udp.log
+}
+
+esp_oui () {
+	get_ouis_from_wifi_log | grep -iqf- oui/esp.txt
+}
+
+non_esp_oui () {
+	get_ouis_from_wifi_log | grep -iqf- oui/nonesp.txt
+}
+
+check_for_common_issues () {
+	if psk_02; then
+		echo "Your device's firmware is too new."
+		echo "Tuya patched the PSK vulnerability that we use to establish a connection."
+		echo "You might still be able to flash this device over serial."
+		echo "For more information and to follow progress on solving this issue see:"
+		echo "https://github.com/ct-Open-Source/tuya-convert/wiki/Collaboration-document-for-PSK-Identity-02"
+		exit 1
+	fi
+	if web_non_esp || udp_non_esp || non_esp_oui; then
+		echo "Your device does not use an ESP82xx."
+		echo "This means you cannot flash custom ESP firmware even over serial."
+		exit 1
+	fi
+	if esp_oui; then
+		echo "An ESP82xx based device connected according to your wifi log."
+		echo "If this is the device you are trying to flash, another issue may be preventing it from flashing."
+		echo "Otherwise, it could be that the device does not use an ESP82xx or it did not connect."
+	else
+		echo "No ESP82xx based devices connected according to your wifi log."
+		echo "Here is a list of all the MAC addresses that connected:"
+		get_macs_from_wifi_log
+		echo
+		echo "If you see your IoT device in this list, it is not an ESP82xx based device."
+		echo "Otherwise, another issue may be preventing it from connecting."
+	fi
+	echo "For additional information, check the *.log files inside the scripts folder."
+	echo "Please include these logs when opening a new issue on our GitHub issue tracker."
+}
+
+check_for_common_issues
+

scripts/oui/esp.txt

@@ -0,0 +1,70 @@
+# "Espressif Inc." OUIs
+# updated Jan 26 2021
+083AF2
+0CDC7E
+10521C
+18FE34
+240AC4
+2462AB
+246F28
+24A160
+24B2DE
+2C3AE8
+2CF432
+30AEA4
+3C6105
+3C71BF
+40F520
+483FDA
+4C11AE
+4C7525
+500291
+545AA6
+5CCF7F
+600194
+6055F9
+68C63A
+70039F
+7C9EBD
+7CDFA1
+807D3A
+840D8E
+84CCA8
+84F3EB
+8CAAB5
+8CCE4E
+9097D5
+943CC6
+94B97E
+98CDAC
+98F4AB
+9C9C1F
+A020A6
+A0764E
+A47B9D
+A4CF12
+A4E57C
+A8032A
+AC67B2
+ACD074
+B4E62D
+B8F009
+BCDDC2
+BCFF4D
+C44F33
+C45BBE
+C4DD57
+C82B96
+CC50E3
+D8A01D
+D8BFC0
+D8F15B
+DC4F22
+E09806
+E0E2E6
+E868E7
+E8DB84
+ECFABC
+F008D1
+F4CFA2
+FCF5C4

scripts/oui/nonesp.txt

@@ -0,0 +1,17 @@
+# Known unsupported OUIs including
+# "Tuya (HK) Limietd"
+# "Tuya Smart Inc."
+# "Beijing Winner Microelectronics Ltd"
+# updated Jan 26 2021
+000D8A
+000F89
+10D561
+1869D8
+286DCD
+68572D
+708976
+807484
+84E342
+D4A651
+DC446D
+E078A3

scripts/setup_ap.sh

@@ -42,6 +42,7 @@ setup () {
 		--interface=$WLAN \
 		--bind-interfaces \
 		--listen-address=$GATEWAY \
+		--except-interface=lo \
 		--dhcp-range=10.42.42.10,10.42.42.40,12h \
 		--address=/#/$GATEWAY
 

start_flash.sh

@@ -80,9 +80,12 @@ while true; do
 		printf .
 		if (( --i == 0 )); then
 			echo
-			echo "Device did not appear with the intermediate firmware"
-			echo "Check the *.log files in the scripts folder"
+			echo "Timed out while waiting for the device to (re)connect"
 			pkill -f smartconfig/main.py && echo "Stopping smart config"
+			echo "======================================================"
+			echo "Attempting to diagnose the issue..."
+			./dr_tuya.sh
+			echo "======================================================"
 			read -p "Do you want to try flashing another device? [y/N] " -n 1 -r
 			echo
 			[[ "$REPLY" =~ ^[Yy]$ ]] || break 2