User Tools

Site Tools


wiki:support:troubleshoot

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
wiki:support:troubleshoot [2021/06/14 09:37]
tda
wiki:support:troubleshoot [2023/06/02 15:47] (current)
ehe [Automatic log gathering]
Line 77: Line 77:
  
 To simplify the log gathering process, two scripts are available. These scripts generate an archive containing many logs and commands results. To simplify the log gathering process, two scripts are available. These scripts generate an archive containing many logs and commands results.
 +
 +++++Gathering logs from WebUI since release 5.9|
 +[[wiki:​systeme_mana:​webui#​gateway_administration_menu|Get logs from gateway]]
 +++++
  
 ++++Gathering logs from shell since release 4.2| ++++Gathering logs from shell since release 4.2|
Line 84: Line 88:
   * It will generate an archive in the same directory you executed the script (example: ''​Logs_2E0605F5_7b26_20200124-095206.tar.gz''​)   * It will generate an archive in the same directory you executed the script (example: ''​Logs_2E0605F5_7b26_20200124-095206.tar.gz''​)
   * Retrieve this archive (use whatever method you prefer: FileZilla, scp, ...).   * Retrieve this archive (use whatever method you prefer: FileZilla, scp, ...).
 +
 +<​note>​**To use only in case of speed or data restriction**,​ there is a new option to gather lighter logs: ''​get_logs -m''​ or ''​get_logs --minimal''​. \\
 +**Please note that it will only gather latest logs** (.log & .log.1 files). \\
 +It will generate an light archive recognizable by its name: _minimal.tar.gz (example: ''​Logs_3903002E_0091_20230419-134556_minimal.tar.gz''​)
 +</​note>​
 ++++ ++++
  
Line 95: Line 104:
   * It will generate an archive in the same directory you executed the script. (example: ''​Logs_2E0605F5_7b26_20200124-095206.tar.gz''​)   * It will generate an archive in the same directory you executed the script. (example: ''​Logs_2E0605F5_7b26_20200124-095206.tar.gz''​)
   * Retrieve this archive (use whatever method you prefer: FileZilla, scp, ...).   * Retrieve this archive (use whatever method you prefer: FileZilla, scp, ...).
 +
  
 <code bash get_logs.sh>​ <code bash get_logs.sh>​
Line 561: Line 571:
 fi fi
 </​code>​ </​code>​
 +
 +
  
 ++++ ++++
Line 573: Line 585:
   * Unplug the USB flash drive.   * Unplug the USB flash drive.
   * An archive with all the required logs has been created on the USB flash drive.   * An archive with all the required logs has been created on the USB flash drive.
 +
  
 <code bash usb.autorun>​ <code bash usb.autorun>​
Line 578: Line 591:
  
 # Version of the script # Version of the script
-SCRIPT_VERSION=1.3+SCRIPT_VERSION=1.4
  
 # Output directory for the result of the script # Output directory for the result of the script
Line 607: Line 620:
 # #
 show_version() { show_version() {
- echo "​Script version: ${SCRIPT_VERSION}"​+        ​echo "​Script version: ${SCRIPT_VERSION}"​
 } }
  
Line 616: Line 629:
 show_help() show_help()
 { {
- show_version +        ​show_version 
- echo -e "​Usage:​ $0 [-h|--help] [-l|--ignore-leds] [-o|--output-dir <​directory>​]+        echo -e "​Usage:​ $0 [-h|--help] [-l|--ignore-leds] [-o|--output-dir <​directory>​]
 \t-h|--help:​ this help \t-h|--help:​ this help
 \t-v|--version:​ version of the script \t-v|--version:​ version of the script
Line 623: Line 636:
 \t-o|--output-dir <​directory>:​ force the output <​directory>​ to use for extraction (default: current directory) \t-o|--output-dir <​directory>:​ force the output <​directory>​ to use for extraction (default: current directory)
 \t-c|--disable-commands:​ disable diagnosis commands (default: executed) \t-c|--disable-commands:​ disable diagnosis commands (default: executed)
 +\t-m|--minimal:​ get minimal logs (to lower generated file size)
 " "
 } }
Line 633: Line 647:
 # #
 compare_versions () { compare_versions () {
- if [[ $1 == $2 ]] +        ​if [[ $1 == $2 ]] 
- then +        then 
- return 0 +                return 0 
- fi +        fi 
- local IFS=. +        local IFS=. 
- local i ver1=($1) ver2=($2) +        local i ver1=($1) ver2=($2) 
- # fill empty fields in ver1 with zeros +        # fill empty fields in ver1 with zeros 
- for ((i=${#​ver1[@]};​ i<​${#​ver2[@]};​ i++)) +        for ((i=${#​ver1[@]};​ i<​${#​ver2[@]};​ i++))do 
- do +                ver1[i]=0 
- ver1[i]=0 +        done 
- done +        for ((i=0; i<​${#​ver1[@]};​ i++))do 
- for ((i=0; i<​${#​ver1[@]};​ i++)) +                if [[ -z ${ver2[i]} ]] 
- do +                then 
- if [[ -z ${ver2[i]} ]] +                        # fill empty fields in ver2 with zeros 
- then +                        ver2[i]=0 
- # fill empty fields in ver2 with zeros +                fi 
- ver2[i]=0 +                if ((10#​${ver1[i]} > 10#​${ver2[i]})) 
- fi +                then 
- if ((10#​${ver1[i]} > 10#​${ver2[i]})) +                        return 1 
- then +                fi 
- return 1 +                if ((10#​${ver1[i]} < 10#​${ver2[i]})) 
- fi +                then 
- if ((10#​${ver1[i]} < 10#​${ver2[i]})) +                        return 2 
- then +                fi 
- return 2 +        done 
- fi +        return 0
- done +
- return 0+
 } }
  
Line 671: Line 683:
 # #
 copy_if_exists(){ copy_if_exists(){
- local from="​$1"​ +        ​local from="​$1"​ 
- local to="​$2"​+        local to="​$2"​
  
- if [ -e "​${from}"​ ] ; then +        ​if [ -e "​${from}"​ ] ; then 
- [ ! -d "​${to}"​ ] && mkdir -p "​${to}"​ +                [ ! -d "​${to}"​ ] && mkdir -p "​${to}"​ 
- cp -a "​${from}"​ "​${to}"​ +                cp -a "​${from}"​ "​${to}"​ 
- fi+        fi
 } }
  
Line 687: Line 699:
 # #
 check_fw_version(){ check_fw_version(){
- # check if /​etc/​version exists +        ​# check if /​etc/​version exists 
- if [ -a /​etc/​version ]; then +        if [ -a /​etc/​version ]; then 
- FW_VERSION=$(cat /​etc/​version)+                FW_VERSION=$(cat /​etc/​version)
  
- # Detect type of firmware: KerOS or SPN +                ​# Detect type of firmware: KerOS or SPN 
- if [[ "​${FW_VERSION}"​ == *-spn-* ]]; then +                if [[ "​${FW_VERSION}"​ == *-spn-* ]]; then 
- # Check if SPN version is not inferior to 2.1, stop if it is the case +                        # Check if SPN version is not inferior to 2.1, stop if it is the case 
- compare_versions ${FW_VERSION:​0:​3} "​2.1"​ +                        compare_versions ${FW_VERSION:​0:​3} "​2.1"​ 
- if [ $? -eq 2 ]; then +                        if [ $? -eq 2 ]; then 
- echo "Your SPN firmware version ($FW_VERSION) is inferior to v2.1. Please use the adequate script."​ >> ${OUTPUT_DIR}/​WRONG_FIRMWARE_VERSION.txt +                                echo "Your SPN firmware version ($FW_VERSION) is inferior to v2.1. Please use the adequate script."​ >> ${OUTPUT_DIR}/​WRONG_FIRMWARE_VERSION.txt 
- exit +                                exit 
- fi +                        fi 
- else +                else 
- # Check if KerOS version is not inferior to 4.0, stop if it is the case +                        # Check if KerOS version is not inferior to 4.0, stop if it is the case 
- compare_versions ${FW_VERSION:​0:​3} "​4.0"​ +                        compare_versions ${FW_VERSION:​0:​3} "​4.0"​ 
- if [ $? -eq 2 ]; then +                        if [ $? -eq 2 ]; then 
- echo "Your KerOS firmware version ($FW_VERSION) is inferior to v4. Please use the adequate script."​ >> ${OUTPUT_DIR}/​WRONG_FIRMWARE_VERSION.txt +                                echo "Your KerOS firmware version ($FW_VERSION) is inferior to v4. Please use the adequate script."​ >> ${OUTPUT_DIR}/​WRONG_FIRMWARE_VERSION.txt 
- exit +                                exit 
- fi +                        fi 
- fi +                fi 
- else +        else 
- echo "Could not find /​etc/​version"​ >> ${OUTPUT_DIR}/​WRONG_FIRMWARE_VERSION.txt +                echo "Could not find /​etc/​version"​ >> ${OUTPUT_DIR}/​WRONG_FIRMWARE_VERSION.txt 
- exit +                exit 
- fi+        fi
 } }
  
Line 718: Line 730:
 # #
 detect_platform() { detect_platform() {
- PLATFORM="​$(fslotpreg getplatform)"​ +        ​PLATFORM="​$(fslotpreg getplatform)"​ 
- if [ $? -ne 0 ]; +        if [ $? -ne 0 ]; 
- then +        then 
- PLATFORM="​NOT_SUPPORTED"​ +                PLATFORM="​NOT_SUPPORTED"​ 
- fi +        fi 
- echo "​Platform detected: ${PLATFORM}"​+        echo "​Platform detected: ${PLATFORM}"​
 } }
  
Line 731: Line 743:
 # #
 fill_serials() { fill_serials() {
- local reg=$(fslotpreg read GP1) +        ​local reg=$(fslotpreg read GP1) 
- if [ $? -eq 0 ]; then +        if [ $? -eq 0 ]; then 
- SERIAL=$(printf "​%X\n"​ $reg) +                SERIAL=$(printf "​%X\n"​ $reg) 
- else +        else 
- SERIAL="​unknown"​ +                SERIAL="​unknown"​ 
- fi +        fi 
- SHORT_MAC=$(cat /​sys/​class/​net/​eth0/​address | awk -F: '​{print $5$6}'​)+        if [ -f "/​sys/​class/​net/​eth0/​address"​ ]; then 
 +                ​SHORT_MAC=$(cat /​sys/​class/​net/​eth0/​address | awk -F: '​{print $5$6}'​) 
 +        elif [ -f "/​sys/​class/​net/​usb0/​address"​ ]; then 
 +                SHORT_MAC=$(cat /​sys/​class/​net/​usb0/​address | awk -F: '​{print $5$6}'​) 
 +        else 
 +                SHORT_MAC="​XXXX"​ 
 +        fi
 } }
  
Line 748: Line 766:
 # #
 detect_env() { detect_env() {
- # Detection is only based on the presence or not of the variable DEVNAME +        ​# Detection is only based on the presence or not of the variable DEVNAME 
- if [ ! -z ${DEVNAME} ] ; then +        if [ ! -z ${DEVNAME} ] ; then 
- SCRIPT_ENV=udev +                SCRIPT_ENV=udev 
- exec > /​dev/​console 2>&1 # Output the script execution in debug console +                exec > /​dev/​console 2>&1 # Output the script execution in debug console 
- else +        else 
- SCRIPT_ENV=shell +                SCRIPT_ENV=shell 
- LEDS_MANAGEMENT=false +                LEDS_MANAGEMENT=false 
- OUTPUT_DIR=$(pwd) +                OUTPUT_DIR=$(pwd) 
- fi +        fi 
- echo "​Script environment:​ ${SCRIPT_ENV}"​+        echo "​Script environment:​ ${SCRIPT_ENV}"​
 } }
  
Line 765: Line 783:
 # #
 detect_leds() { detect_leds() {
- if [ -e "​${SYSLEDPATH}"​ ]; then +        ​if [ -e "​${SYSLEDPATH}"​ ]; then 
- LED_PATH="​${SYSLEDPATH}"​ +                LED_PATH="​${SYSLEDPATH}"​ 
- elif [ -e "​${SYSLEDPATH_4_0_IBTS}"​ ]; then +        elif [ -e "​${SYSLEDPATH_4_0_IBTS}"​ ]; then 
- LED_PATH="​${SYSLEDPATH_4_0_IBTS}"​ +                LED_PATH="​${SYSLEDPATH_4_0_IBTS}"​ 
- elif [ -e "​${SYSLEDPATH_4_0_IFEMTO}"​ ]; then +        elif [ -e "​${SYSLEDPATH_4_0_IFEMTO}"​ ]; then 
- LED_PATH="​${SYSLEDPATH_4_0_IFEMTO}"​ +                LED_PATH="​${SYSLEDPATH_4_0_IFEMTO}"​ 
- fi+        fi
 } }
  
Line 782: Line 800:
 # #
 set_led() { set_led() {
- trigger="​${1}"​ +        ​trigger="​${1}"​ 
- on="​$2"​ +        on="​$2"​ 
- off="​$3"​+        off="​$3"​
  
- if [ -e "​${LED_PATH}"​ ]; then +        ​if [ -e "​${LED_PATH}"​ ]; then 
- echo $trigger > "​${LED_PATH}/​trigger"​ +                echo $trigger > "​${LED_PATH}/​trigger"​ 
- case "​$trigger"​ in +                case "​$trigger"​ in 
- "​timer"​) +                        "​timer"​) 
- echo ${on:=500} > "​${LED_PATH}/​delay_on"​ +                                echo ${on:=500} > "​${LED_PATH}/​delay_on"​ 
- echo ${off:=500} > "​${LED_PATH}/​delay_off"​ +                                echo ${off:=500} > "​${LED_PATH}/​delay_off"​ 
- ;; +                                ;; 
- "​none"​) +                        "​none"​) 
- echo ${on:=0} > "​${LED_PATH}/​brightness"​ +                                echo ${on:=0} > "​${LED_PATH}/​brightness"​ 
- ;; +                                ;; 
- *) +                        *) 
- # Nothing to do +                                # Nothing to do 
- ;; +                                ;; 
- esac +                esac 
- fi+        fi
 } }
  
 get_nmea_frames() { get_nmea_frames() {
- gps_tty="​$1"​+        ​gps_tty="​$1"​
  
- if ! [ -e "​$gps_tty"​ ]; then +        ​if ! [ -e "​$gps_tty"​ ]; then 
- echo "​$gps_tty doesn'​t exist"​ +                echo "​$gps_tty doesn'​t exist"​ 
- return 1 +                return 1 
- fi+        fi
  
- if [ -n "​$(lsof | grep $(readlink $gps_tty))"​ ]; then +        ​if [ -n "​$(lsof | grep $(readlink $gps_tty))"​ ]; then 
- echo "​$gps_tty already used by $(lsof | grep $(readlink $gps_tty))"​ +                echo "​$gps_tty already used by $(lsof | grep $(readlink $gps_tty))"​ 
- return 2 +                return 2 
- else +        else 
- # Pseudo-term keep only FIRST frames so we keep last of these old frames and get few new frames +                # Pseudo-term keep only FIRST frames so we keep last of these old frames and get few new frames 
- # It will take 3 seconds +                # It will take 3 seconds 
- timeout -t 3 cat $gps_tty > /​tmp/​nmea.log +                timeout -t 3 cat $gps_tty > /​tmp/​nmea.log 
- cat /​tmp/​nmea.log +                cat /​tmp/​nmea.log 
- fi+        fi
 } }
  
Line 826: Line 844:
 # #
 execute_commands() { execute_commands() {
- echo "​Executing diagnosis commands (can take a long time)"+        ​echo "​Executing diagnosis commands (can take a long time)"
  
- echo -e "​\tSystems commands"​ +        ​echo -e "​\tSystems commands"​ 
- +                
- echo -e "​\n\n\n************** ​ fslotpreg getplatform ​ *****************"​ +                echo -e "​\n\n\n************** ​ fslotpreg getplatform ​ *****************"​ 
- fslotpreg getplatform +                fslotpreg getplatform 
- echo -e "​\n\n\n************** ​ fslotpreg fslotpreg read GP1  *****************"​ +                echo -e "​\n\n\n************** ​ fslotpreg fslotpreg read GP1  *****************"​ 
- fslotpreg read GP1 +                fslotpreg read GP1 
- echo -e "​\n\n\n************** ​ date  *****************"​ +                echo -e "​\n\n\n************** ​ date  *****************"​ 
- date +                date 
- echo -e "​\n\n\n************** ​ uptime ​ *****************"​ +                echo -e "​\n\n\n************** ​ uptime ​ *****************"​ 
- uptime +                uptime 
- echo -e "​\n\n\n************** ​ ps ww *****************"​ +                echo -e "​\n\n\n************** ​ ps ww *****************"​ 
- ps ww +                ps ww 
- echo -e "​\n\n\n************** ​ opkg list-installed ​ ​*****************"​ +                echo -e "​\n\n\n************** ​ opkg status ​ ​*****************"​ 
- opkg list-installed +                opkg status 
- echo -e "​\n\n\n**************  ​opkg status ​ *****************"​ +                echo -e "​\n\n\n**************  ​monit status ​ *****************"​ 
- opkg status +                ​monit ​status 
- echo -e "​\n\n\n**************  ​monit status ​ ​*****************"​ +                echo -e "​\n\n\n**************  ​mount  ​*****************"​ 
- monit status +                mount 
- echo -e "​\n\n\n**************  ​mount  ​*****************"​ +                echo -e "​\n\n\n**************  ​OpenVPN Certificate expiration date  ​*****************"​ 
- mount +                ​compare_versions ${FW_VERSION:​0:​3} "​5.0"​ 
- echo -e "​\n\n\n**************  ​ztp  ​*****************"​ +                ret=$? 
- cat /​etc/​ztp/​status.ini +                if [ $ret -eq 1 ]; then 
- cat /etc/certs/*.csr +                        if [ -f "/​etc/​openvpn/​bscc.p12"​ ]; then 
- cat /etc/certs/ztp.pem+                                openssl pkcs12 -info -in /​etc/​openvpn/​bscc.p12 -nokeys -password file:/​etc/​openvpn/​bscc.password| openssl x509 -noout -enddate 
 +                        fi 
 +                elif [ $ret -eq 2 ]; then 
 +                        grep -iA3 "​Validity"​ /​.update/​update.log | tail -3 | grep -i "not after" | sed 's/^[ \t]*//'​ 
 +                fi 
 +                echo -e "​\n\n\n**************  ​Disk info (df -h)  ​*****************"​ 
 +                df -h 
 +                echo -e "​\n\n\n************** ​ Log Size (du -sh /var/log)  ​*****************"​ 
 +                du -sh /var/log 
 +        ) >> ${tmpdir}/cmd_results.txt
  
- ) >> ${tmpdir}/cmd_results.txt+        echo -e "​\tNetwork commands"​ 
 +        ( 
 +                echo -e "​\n\n\n************** ​ ifconfig -a  *****************"​ 
 +                ifconfig -a 
 +                echo -e "​\n\n\n************** ​ gsmdiag.py ​ *****************"​ 
 +                gsmdiag.py ; cat /tmp/gsmdiag.txt 
 +                echo -e "​\n\n\n************** ​ route  *****************"​ 
 +                route -n 
 +                echo -e "​\n\n\n************** ​ iptables -L  *****************"​ 
 +                iptables -L 
 +                echo -e "​\n\n\n************** ​ connmanctl technologies ​ *****************"​ 
 +                connmanctl technologies 
 +                echo -e "​\n\n\n************** ​ services ​ *****************"​ 
 +                connmanctl services
  
- echo -e "​\tNetwork commands"​ +                if [ -e /​dev/​cdc-wdm1 ]; then 
- ( +                        echo -e "​\n\n\n**************  ​qmicli get mode  ​*****************"​ 
- echo -e "​\n\n\n**************  ​ifconfig -a  ​*****************"​ +                        ​qmicli ​-/dev/cdc-wdm1 --nas-get-system-selection-preference 
- ifconfig ​-+                fi 
- echo -e "​\n\n\n************** ​ gsmdiag.py ​ *****************"​ +        ) >> ${tmpdir}/​network_result.txt
- gsmdiag.py ; cat /tmp/gsmdiag.txt +
- echo ​-e "​\n\n\n************** ​ route  *****************"​ +
- route ​-+
- echo ​-e "​\n\n\n************** ​ iptables ​-L  *****************"​ +
- iptables ​-+
- echo ​-e "​\n\n\n************** ​ connmanctl technologies ​ *****************"​ +
- connmanctl technologies +
- echo -e "​\n\n\n************** ​ services ​ *****************"​ +
- connmanctl services+
  
- if [ -e /dev/cdc-wdm1 ]; then +        get_nmea_frames ​/dev/nmea3 > ${tmpdir}/​nmea.log 
- echo -e "​\n\n\n************** ​ qmicli get mode  *****************"​ +        [ $? -ne 0 && get_nmea_frames ​/dev/nmea2 >> ${tmpdir}/nmea.log
- qmicli -d /dev/cdc-wdm1 --nas-get-system-selection-preference +
- fi +
->> ${tmpdir}/network_result.txt+
  
- get_nmea_frames ​/dev/nmea3 > ${tmpdir}/​nmea.log +        # Overlay modifications 
- [ $? -ne 0 ] && get_nmea_frames ​/dev/nmea2 >> ${tmpdir}/nmea.log+        echo -e "​\tOverlay commands disabled (KEROS-1907)"​ 
 +#       ​overlay diff -l /.rootfs.ro ​-/user/​.rootfs_upper/ >> ${tmpdir}/overlay_status.txt << EOF 
 +#yes 
 +#EOF
  
- # Overlay modifications +        ​echo "End of diagnosis commands"​
- echo -e "​\tOverlay commands"​ +
- overlay diff -l /.rootfs.ro -u /​user/​.rootfs_upper/​ >> ${tmpdir}/​overlay_status.txt << EOF +
-yes +
-EOF +
- +
- echo "End of diagnosis commands"​+
 } }
  
Line 897: Line 922:
 get_logs() get_logs()
 { {
- local output_dir="​$1"​ +        ​local output_dir="​$1"​ 
- local name="​Logs_${SERIAL}_${SHORT_MAC}_$(date "​+%Y%m%d-%H%M%S"​)"​ +        local option="​$2"​ 
- local tmpdir="/​user/​${name}"​+        ​local name="​Logs_${SERIAL}_${SHORT_MAC}_$(date "​+%Y%m%d-%H%M%S"​)"​ 
 +        if [ "​${option}"​ = "​minimal"​ ]; then 
 +                name="​${name}_minimal"​ 
 +        fi 
 +        ​local tmpdir="/​user/​${name}"​
  
- # tmp files +        ​# tmp files 
- echo "​Extracting tmp files"​ +        echo "​Extracting tmp files"​ 
- copy_if_exists /​tmp/​sys_startup_status.json ${tmpdir}/​tmp/​ +        copy_if_exists /​tmp/​sys_startup_status.json ${tmpdir}/​tmp/​ 
- copy_if_exists /​tmp/​board_info.json ${tmpdir}/​tmp/​ +        copy_if_exists /​tmp/​board_info.json ${tmpdir}/​tmp/​ 
- copy_if_exists /​tmp/​calib_loraloc.json ${tmpdir}/​tmp/​ +        copy_if_exists /​tmp/​calib_loraloc.json ${tmpdir}/​tmp/​ 
- copy_if_exists /​tmp/​calib_rf.json ${tmpdir}/​tmp/​+        copy_if_exists /​tmp/​calib_rf.json ${tmpdir}/​tmp/​
  
- # network configuration +        ​# network configuration 
- echo "​Extracting network configuration"​ +        echo "​Extracting network configuration"​ 
- copy_if_exists /​etc/​network/​ ${tmpdir}/​etc/​ +        if [ "​${option}"​ = "​minimal"​ ]; then 
- copy_if_exists /​etc/​firewall.d/​ ${tmpdir}/​etc/​+                ​copy_if_exists /​etc/​network/connman/ ${tmpdir}/​etc/​network/ 
 +                copy_if_exists /​etc/​network/​ofono/​ ${tmpdir}/​etc/​network/​ 
 +                copy_if_exists /​etc/​network/​openvpn/​ ${tmpdir}/​etc/​network/​ 
 +                cp /​etc/​network/​*.conf ${tmpdir}/​etc/​network/​ 
 +        else 
 +                copy_if_exists /​etc/​network/​ ${tmpdir}/​etc/​ 
 +        fi 
 +        copy_if_exists /​etc/​firewall.d/​ ${tmpdir}/​etc/​
  
- CPF (lorad + lorafwd) ​configuration +        ​init configuration 
- echo "​Extracting CPF configuration (if any)" +        if "${option}" != "​minimal" ​]; then 
- for lorad_dir in /​user/​etc/​lorad /​etc/​lorad +                echo "Extracting init.d and rc*.d files
- do +                copy_if_exists /etc/init.d/ ${tmpdir}/​etc/​ 
- if [ -d ${lorad_dir} ] ; then +                for filename ​in /etc/rc*.d; ​do 
- echo -e "\tExtracting lorad configuration+                        ​copy_if_exists ​${filename} ${tmpdir}/​etc/​ 
- copy_if_exists /etc/default/lorad ${tmpdir}/​etc/​default/ +                done 
- copy_if_exists ${lorad_dir} ${tmpdir}${lorad_dir%/​*} +        fi
- fi +
- done +
- for lorafwd_dir ​in /user/etc/lorafwd /​etc/​lorafwd +
- do +
- if [ -d ${lorafwd_dir] ; then +
- echo -e "​\tExtracting lorafwd configuration"​ +
- copy_if_exists /​etc/​default/​lorafwd ​${tmpdir}/​etc/default+
- copy_if_exists ${lorafwd_dir} ${tmpdir}${lorafwd_dir%/​*} +
- fi +
- done+
  
- WMC configuration ​and logs +        ​CPF (lorad + lorafwd) ​configuration 
- echo "​Extracting ​WMC configuration ​and logs (if any)"​ +        echo "​Extracting ​CPF configuration (if any)"​ 
- if [ -d /​user/​bscc ​] ; then +        if [ "​${option}"​ = "​minimal" ​]; then 
- echo -e "​\tExtracting BSCC configuration and logs"​ +                copy_if_exists /user/etc/lorad/lorad.json ${tmpdir}/​user/​etc/lorad
- copy_if_exists /user/bscc/var.xml ${tmpdir}/​user/​bscc+                copy_if_exists /etc/lorad/lorad.json ​${tmpdir}/etc/lorad
- copy_if_exists /user/bscc/traces/ ${tmpdir}/user/bscc+        else 
- fi +                for lorad_dir in /​user/​etc/​lorad /etc/lorad; do 
- if [ -d /​user/​snmp ​] ; then +                        ​if [ -d ${lorad_dir} ​] ; then 
- echo -e "​\tExtracting ​SNMP configuration ​and logs+                                echo -e "​\tExtracting ​lorad configuration"​ 
- copy_if_exists ​/​user/​snmp/​snmpd.conf ​${tmpdir}/user/snmp/ +                                copy_if_exists ​${lorad_dir} ​${tmpdir}${lorad_dir%/*} 
- copy_if_exists /user/snmp/traces/ ${tmpdir}/user/snmp/ +                        fi 
- fi+                done 
 +        fi 
 +        copy_if_exists /etc/default/lorad ${tmpdir}/etc/default/
  
- # miscellaneous configuration and logs +        ​echo -e "\tExtracting lorafwd ​configuration"​ 
- echo "Extracting miscellaneous ​configuration ​and logs+        if [ "${option}" = "​minimal"​ ]; then 
- copy_if_exists /​etc/​rcU.d/ ​${tmpdir}/etc/ +                for filename in /etc/lorafwd/*.toml; do 
- copy_if_exists ​/.update/​update.log ${tmpdir}/ +                        copy_if_exists ${filename} ${tmpdir}/etc/lorafwd/ 
- copy_if_exists ​/​user/​.update ​${tmpdir}/user/ +                done 
- copy_if_exists /​user/​.upgrade.log ​${tmpdir}/user+        else 
- copy_if_exists ​/user/.sysupgrade.log ${tmpdir}/user/ +                for lorafwd_dir in /user/etc/lorafwd ​/etc/lorafwd; do 
- copy_if_exists ​/etc/version ​${tmpdir}/etc/ +                        if [ -d ${lorafwd_dir] ; then 
- copy_if_exists ​/​etc/​sysupgrade.conf ​${tmpdir}/etc/ +                                copy_if_exists ​${lorafwd_dir} ​${tmpdir}${lorafwd_dir%/*} 
- copy_if_exists /etc/sysupgrade.d/ ${tmpdir}/​etc/​ +                        fi 
- copy_if_exists /etc/.squash_layer_version ​${tmpdir}/​etc/​ +                done 
- echo ${SCRIPT_VERSION} > ${tmpdir}/script_version+        fi 
 +        copy_if_exists /etc/lorafwd.toml ${tmpdir}/​etc/​ 
 +        copy_if_exists /etc/default/​lorafwd ​${tmpdir}/​etc/​default/
  
- if ${EXEC_COMMANDS} ; then +        # WMC configuration and logs 
- execute_commands +        echo "​Extracting WMC configuration and logs (if any)"​ 
- fi+        if [ -d /user/bscc ] ; then 
 +                echo -e "​\tExtracting BSCC configuration and logs"​ 
 +                copy_if_exists /​user/​bscc/​var.xml ​${tmpdir}/​user/​bscc/​ 
 +                copy_if_exists /​user/​bscc/​traces/​ ${tmpdir}/​user/​bscc/​ 
 +        fi 
 +        if [ -d /user/snmp ] ; then 
 +                echo -e "​\tExtracting SNMP configuration and logs" 
 +                ​copy_if_exists /​user/​snmp/​snmpd.conf ${tmpdir}/​user/​snmp/​ 
 +                copy_if_exists /​user/​snmp/​traces/​ ${tmpdir}/​user/​snmp/​ 
 +        ​fi 
 +        copy_if_exists /​etc/​default/​bscc ${tmpdir}/​etc/​default/​ 
 +        copy_if_exists /​etc/​default/​lora-snmp ${tmpdir}/​etc/​default/​ 
 +        copy_if_exists /​etc/​snmp/​snmpd.conf ${tmpdir}/​etc/​snmp/​
  
- Finish by system ​logs to prevent space errors +        ​basic station configuration and logs 
- echo "​Extracting ​system ​logs"​ +        echo "​Extracting ​basic station configuration and logs (if any)
- copy_if_exists /var/log/ ${tmpdir}/var/+        if [ -d /​user/​basic_station ] ; then 
 +                echo -e "​\tExtracting Basic Station configuration and logs"​ 
 +                ​copy_if_exists /user/​basic_station/​etc/​ ${tmpdir}/​user/​basic_station/​ 
 +                copy_if_exists /​user/​basic_station/log/ ${tmpdir}/user/​basic_station/​ 
 +        fi 
 +        copy_if_exists /​etc/​station/​ ${tmpdir}/​etc/
  
- echo "Saving all data in ${output_dir}/${name}.tar.gz" +        # miscellaneous configuration and logs 
- tar -C /user -czf ${output_dir}/${name}.tar.gz ${name+        ​echo "Extracting miscellaneous configuration and logs"​ 
- echo "​Done"​ +        copy_if_exists /​etc/​default/ ​${tmpdir}/etc/ 
- sync+        copy_if_exists /​etc/​openvpn/ ${tmpdir}/etc/ 
 +        copy_if_exists /.update/​update.log ${tmpdir}/ 
 +        ​copy_if_exists ​/user/​initrd/​initrd.log ​${tmpdir}/user/​initrd 
 +        copy_if_exists /​user/​.update ​${tmpdir}/user/ 
 +        copy_if_exists /user/.upgrade.log ${tmpdir}/user/ 
 +        ​copy_if_exists /​user/​.sysupgrade.log ${tmpdir}/​user/​ 
 +        ​copy_if_exists /​etc/​version ${tmpdir}/​etc/​ 
 +        copy_if_exists /​etc/​sysupgrade.conf ${tmpdir}/​etc/​ 
 +        copy_if_exists /​etc/​sysupgrade.d/​ ${tmpdir}/​etc/​ 
 +        copy_if_exists /​etc/​.squash_layer_version ${tmpdir}/​etc/​ 
 +        echo ${SCRIPT_VERSION} > ${tmpdir}/​script_version
  
- rm -rf $tmpdir+        if ${EXEC_COMMANDS} ; then 
 +                execute_commands 
 +        fi 
 + 
 +        # Finish by system logs to prevent space errors 
 +        echo "​Extracting system logs"​ 
 +        if [ "​${option}"​ = "​minimal"​ ]; then 
 +                copy_if_exists /​var/​log/​messages ${tmpdir}/​var/​log/​ 
 +                copy_if_exists /​var/​log/​dmesg ${tmpdir}/​var/​log/​ 
 +                for filename in /​var/​log/​*.1;​ do 
 +                        copy_if_exists ${filename} ${tmpdir}/​var/​log/​ 
 +                done 
 +                for filename in /​var/​log/​*.log;​ do 
 +                        copy_if_exists ${filename} ${tmpdir}/​var/​log/​ 
 +                done 
 +        else 
 +                copy_if_exists /var/log/ ${tmpdir}/​var/​ 
 +        fi 
 + 
 +        echo "​Saving all data in ${output_dir}/​${name}.tar.gz"​ 
 +        tar -C /user -czf ${output_dir}/​${name}.tar.gz ${name} 
 +        echo "​Done"​ 
 +        sync 
 + 
 +        ​rm -rf $tmpdir
 } }
  
Line 987: Line 1068:
 key="​$1"​ key="​$1"​
  
- case $key in +        ​case $key in 
- -l|--ignore-leds) +                -l|--ignore-leds) 
- LEDS_MANAGEMENT=false +                LEDS_MANAGEMENT=false 
- shift +                shift 
- ;; +                ;; 
- -o|--output-dir) +                -o|--output-dir) 
- OUTPUT_DIR="​$2"​ +                OUTPUT_DIR="​$2"​ 
- shift 2 +                shift 2 
- ;; +                ;; 
- -c|--disable-commands) +                -c|--disable-commands) 
- EXEC_COMMANDS=false +                EXEC_COMMANDS=false 
- shift +                shift 
- ;; +                ;; 
- -h|--help) +                -h|--help) 
- show_help +                show_help 
- shift +                shift 
- exit 1 +                exit 1 
- ;; +                ;; 
- -v|--version) +                -v|--version) 
- show_version +                show_version 
- shift +                shift 
- exit 1 +                exit 1 
- ;; +                ;; 
- *) +                ​-m|--minimal) 
- shift +                OPTION="​minimal"​ 
- ;; +                shift 
- esac+                ;; 
 +                ​*) 
 +                shift 
 +                ;; 
 +        esac
 done done
  
Line 1027: Line 1112:
 # Set led On # Set led On
 if ${LEDS_MANAGEMENT} ; then if ${LEDS_MANAGEMENT} ; then
- detect_leds +        ​detect_leds 
- set_led none 1+        set_led none 1
 fi fi
  
 # Get logs # Get logs
-get_logs ${OUTPUT_DIR}+get_logs ${OUTPUT_DIR} ${OPTION}
  
 # End of script, manage user's notifications # End of script, manage user's notifications
 if ${LEDS_MANAGEMENT} ; then if ${LEDS_MANAGEMENT} ; then
- # Prepare reboot after USB unplug +        ​# Prepare reboot after USB unplug 
- cat > /​tmp/​usbkey_remove << EOF+        cat > /​tmp/​usbkey_remove << EOF
 # Switch off Led # Switch off Led
 echo "​none"​ > ${LED_PATH}/​trigger echo "​none"​ > ${LED_PATH}/​trigger
 EOF EOF
  
- # All is right, make Led blinking fast to notify User +        ​# All is right, make Led blinking fast to notify User 
- set_led timer 100 100+        set_led timer 100 100
 fi fi
 </​code>​ </​code>​
  
 ++++ ++++
 +
 +===== Helium =====
 +This section contains some tips to do the first level of analysis (support level 1) on Helium gateways.\\
 +It will help you to investigate before escalate to Kerlink support (level 2).
 +
 +
 +**Please note that there is an existing open wiki for Helium users: [[https://​wikikerlink.fr/​helium|Kerlink Helium Hotspot Wiki]]**\\
 +  * First step is to follow [[https://​wikikerlink.fr/​helium/​doku.php?​id=home#​troubleshooting_guide|Troubleshooting guide]] chapter which contains:
 +        * [[https://​wikikerlink.fr/​helium/​doku.php?​id=home#​self_diagnostic|Self diagnostic]] section
 +        * [[https://​wikikerlink.fr/​helium/​doku.php?​id=home#​useful_links|Useful links]] section
 +
 +  * Second step, if necessary, is to follow the table below:
 +
 +^ Source of info ^ KPI ^ Value ^ Action ^
 +^ [[https://​helium-onboarding.kerlink.com/​|Onboarding dashboard]] | Last seen | More than 30m ago | Gateway is offline. Ask customer to check power/​internet. |
 +| ::: | ::: | Less than 30m ago | Gateway is online. Check next KPI. |
 +| ::: | Block height | = 0 | Gateway is downloading a snapshot. |
 +| ::: | ::: | = 1 | Gateway is loading the downloaded snapshot. |
 +| ::: | ::: | Else | KPI is ok, check next KPI. |
 +| ::: | Block age | > 30 mn | Gateway is syncing, check again in one hour. If it persists, you can open a ticket to Kerlink Support. |
 +| ::: | ::: | < 30 mn | Gateway is synced. Check next KPI. |
 +^ [[https://​explorer.helium.com//​|Helium explorer]] | 24h/14d earnings | > 0 HNT | Seems OK. Confirm with other Explorer KPIs. |
 +| ::: | Last activity date | < 3h | Seems OK. Confirm with other Explorer KPIs. |
 +| ::: | Is there a constructed challenge in the past 24h? | Yes | Seems OK. Confirm with other Explorer KPIs. |
 +| ::: | ::: | No | When gateway is synced, it should challenge. You can open a ticket to Kerlink Support. |
 +| ::: | Is there a challenged beaconer in the past 24h? | Yes | Seems OK. Confirm with other Explorer KPIs. |
 +| ::: | ::: | No | Gateway might be relayed. If not, you can open a ticket to Kerlink Support. |
 +| ::: | Is there a witnessed beacon in the past 8-24h? | Value depends on density around miner | Seems OK. Confirm with other Explorer KPIs. |
 +| ::: | ::: | No | You can open a ticket to Kerlink Support. |
 +| ::: | Is there a “broadcasted beacon” in the past 5d? | Yes | Seems OK. Confirm with other Explorer KPIs. |
 +| ::: | ::: | No | This is random but it seems odd. You can open a ticket to Kerlink Support. |
 +
 +==== Gateways connected to a WMC====
 +For gateways connected to a WMC, you can use the Remote Command: Remote Access > Command\\
 +{{:​images:​remote_command_wanesy_management_center.png?​800|}}
 +<note important>​** Due its cryptographic key and to ensure Kerlink Helium Hotspot security, no SSH access can be given to those gateways. \\     ​**</​note>​
 +
 +You can use commands below:
 +^ Command ^ Awaited result ^ 
 +| opkg list | {{:​images:​opkg.png?​400|}} |
 +| monit status miner | {{:​images:​monit_status_miner.png?​400|}} |Status = OK | 
 +| monit status lorafwd_helium | {{:​images:​monit_status_lorafwd_helium.png?​400|}} | Status = OK |
 +| /​etc/​init.d/​miner status | {{:​images:​miner_status.png?​400|}} | Check Diff |
 +
 +If the result is different than the awaited result, you can open a ticket to Kerlink Support.
wiki/support/troubleshoot.1623656266.txt.gz · Last modified: 2021/06/14 09:37 by tda