wiki:support:troubleshoot
Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
|
wiki:support:troubleshoot [2019/09/25 15:57] ghi created |
wiki:support:troubleshoot [2023/06/02 15:47] (current) ehe [Automatic log gathering] |
||
|---|---|---|---|
| Line 8: | Line 8: | ||
| * ''/var/log/messages*'': These files are the main logs, they gather the traces of the firmware. They contain the boot traces, the network events, the hardware events, and many other traces. | * ''/var/log/messages*'': These files are the main logs, they gather the traces of the firmware. They contain the boot traces, the network events, the hardware events, and many other traces. | ||
| - | * ''/var/log/lora.log*'': Theses files contains the traces of the common packet forwarder (lorad and lorafwd). If you encounter LoRa issues, this is a good way to start. The verbosity of this log can be increased/decreased. Refer to the [[ wiki:cpf#log_managment |log management]] section. | + | * ''/var/log/lora.log*'': Theses files contain the traces of the common packet forwarder (lorad and lorafwd). If you encounter LoRa issues, this is a good way to start. The verbosity of this log can be increased/decreased. Refer to the [[ wiki:lora:cpf#log_management |log management]] section. |
| * ''/tmp/board_info.json'': This file contains information about the hardware of the gateway. It is generated at each boot time.\\ | * ''/tmp/board_info.json'': This file contains information about the hardware of the gateway. It is generated at each boot time.\\ | ||
| * ''/tmp/calib_loraloc.json'' (**for Wirnet iBTS**): This file contains the calibration values specific to this gateway. It is generated at boot time. | * ''/tmp/calib_loraloc.json'' (**for Wirnet iBTS**): This file contains the calibration values specific to this gateway. It is generated at boot time. | ||
| - | * ''/tmp/calib_rf.json'' (**for Wirnet iFemtoCell and Wirnet iStation**): This file contains the calibration values specific to this gateway. It is generated at boot time. | + | * ''/tmp/calib_rf.json'' (**for Wirnet iFemtoCell, Wirnet iFemtoCell-evolution and Wirnet iStation**): This file contains the calibration values specific to this gateway. It is generated at boot time. |
| * ''/dev/nmea*'' (**Wirnet iBTS and Wirnet iStation**): These devices receive NMEA frames (GPS) every seconds. It can be monitored with the ''cat'' command. | * ''/dev/nmea*'' (**Wirnet iBTS and Wirnet iStation**): These devices receive NMEA frames (GPS) every seconds. It can be monitored with the ''cat'' command. | ||
| Line 66: | Line 66: | ||
| * For Wirnet iFemtoCell, the serial number is written on the rear sticker (it is named "Final product ID"). | * For Wirnet iFemtoCell, the serial number is written on the rear sticker (it is named "Final product ID"). | ||
| * For Wirnet iStation, the serial number is written on the external sticker (it is named "Product ID"). | * For Wirnet iStation, the serial number is written on the external sticker (it is named "Product ID"). | ||
| + | * For Wirnet iFemtoCell-evolution, the serial number is written on rear sticker (it is named "Product ID"). | ||
| * A precise description of your problem. "Gateway not working" is not enough to investigate. | * A precise description of your problem. "Gateway not working" is not enough to investigate. | ||
| * The list of the actions you took to troubleshoot/investigate your problem. | * The list of the actions you took to troubleshoot/investigate your problem. | ||
| Line 77: | Line 78: | ||
| To simplify the log gathering process, two scripts are available. These scripts generate an archive containing many logs and commands results. | To simplify the log gathering process, two scripts are available. These scripts generate an archive containing many logs and commands results. | ||
| - | ++++Gathering logs from shell| | + | ++++Gathering logs from WebUI since release 5.9| |
| + | [[wiki:systeme_mana:webui#gateway_administration_menu|Get logs from gateway]] | ||
| + | ++++ | ||
| + | |||
| + | ++++Gathering logs from shell since release 4.2| | ||
| + | |||
| + | To gather the logs from shell: | ||
| + | * Execute the command ''get_logs''. //The execution of the script takes less than 1 minute.// | ||
| + | * It will generate an archive in the same directory you executed the script (example: ''Logs_2E0605F5_7b26_20200124-095206.tar.gz'') | ||
| + | * Retrieve this archive (use whatever method you prefer: FileZilla, scp, ...). | ||
| + | |||
| + | <note>**To use only in case of speed or data restriction**, there is a new option to gather lighter logs: ''get_logs -m'' or ''get_logs --minimal''. \\ | ||
| + | **Please note that it will only gather latest logs** (.log & .log.1 files). \\ | ||
| + | It will generate an light archive recognizable by its name: _minimal.tar.gz (example: ''Logs_3903002E_0091_20230419-134556_minimal.tar.gz'') | ||
| + | </note> | ||
| + | ++++ | ||
| + | |||
| + | ++++Gathering logs from shell on older releases| | ||
| To gather the logs from shell: | To gather the logs from shell: | ||
| * Upload the script bellow (use whatever method you prefer: FileZilla, scp, copy/paste, ...) in a folder of your gateway and go to this folder. | * Upload the script bellow (use whatever method you prefer: FileZilla, scp, copy/paste, ...) in a folder of your gateway and go to this folder. | ||
| - | * Convert the file to Unix text file format using ''dos2unix gather_logs.sh''. | + | * Convert the file to Unix text file format using ''dos2unix get_logs.sh''. |
| - | * Add execution rights to the script using ''chmod 755 gather_logs.sh''. | + | * Add execution rights to the script using ''chmod 755 get_logs.sh''. |
| - | * Execute the script using ''./gather_logs.sh''. //The execution of the script takes about 1-2 minutes.// | + | * Execute the script using ''./get_logs.sh''. //The execution of the script takes less than 1 minute.// |
| - | * It will generate an archive under ''/tmp''. | + | * It will generate an archive in the same directory you executed the script. (example: ''Logs_2E0605F5_7b26_20200124-095206.tar.gz'') |
| * Retrieve this archive (use whatever method you prefer: FileZilla, scp, ...). | * Retrieve this archive (use whatever method you prefer: FileZilla, scp, ...). | ||
| - | <code bash gather_logs.sh> | + | |
| + | <code bash get_logs.sh> | ||
| #!/bin/sh | #!/bin/sh | ||
| - | USBDIR="`dirname $0`" | + | # Version of the script |
| + | SCRIPT_VERSION=1.3 | ||
| - | SYSLEDPATH_IBTS="/sys/class/leds/yellow:debug/" | + | # Output directory for the result of the script |
| - | SYSLEDPATH_IFEMTO="/sys/class/leds/led1:red:power/" | + | OUTPUT_DIR="`dirname $0`" |
| - | check_version(){ | + | # Execution's environment of the script |
| + | SCRIPT_ENV=unknown | ||
| + | |||
| + | # PLATFORM: IBTS, IFEMTO, ISTATION, IFEVO | ||
| + | PLATFORM=unknown | ||
| + | # Serial number of the product: 39020003 | ||
| + | SERIAL=unknown | ||
| + | # ETH0 MAC: 70:76:ff:01:77:71 => 7771 | ||
| + | SHORT_MAC=unknown | ||
| + | |||
| + | # LED paths | ||
| + | SYSLEDPATH_4_0_IBTS="/sys/class/leds/yellow:debug" | ||
| + | SYSLEDPATH_4_0_IFEMTO="/sys/class/leds/led1:red:power" | ||
| + | SYSLEDPATH="/sys/class/leds/status" | ||
| + | LED_PATH="/do/not/exist" | ||
| + | |||
| + | # Script arguments | ||
| + | LEDS_MANAGEMENT=true | ||
| + | EXEC_COMMANDS=true | ||
| + | |||
| + | # | ||
| + | # Print the version of the script | ||
| + | # | ||
| + | show_version() { | ||
| + | echo "Script version: ${SCRIPT_VERSION}" | ||
| + | } | ||
| + | |||
| + | |||
| + | # | ||
| + | # Show help | ||
| + | # | ||
| + | show_help() | ||
| + | { | ||
| + | show_version | ||
| + | echo -e "Usage: $0 [-h|--help] [-l|--ignore-leds] [-o|--output-dir <directory>] | ||
| + | \t-h|--help: this help | ||
| + | \t-v|--version: version of the script | ||
| + | \t-l|--ignore-leds: this script does not manage the leds (default: autodetection) | ||
| + | \t-o|--output-dir <directory>: force the output <directory> to use for extraction (default: current directory) | ||
| + | \t-c|--disable-commands: disable diagnosis commands (default: executed) | ||
| + | " | ||
| + | } | ||
| + | |||
| + | # | ||
| + | # Compare 2 versions to know if inferior or superior | ||
| + | # Res: 0: $1 = $2 | ||
| + | # 1: $1 > $2 | ||
| + | # 2: $1 < $2 | ||
| + | # | ||
| + | compare_versions () { | ||
| + | if [[ $1 == $2 ]] | ||
| + | then | ||
| + | return 0 | ||
| + | fi | ||
| + | local IFS=. | ||
| + | local i ver1=($1) ver2=($2) | ||
| + | # fill empty fields in ver1 with zeros | ||
| + | for ((i=${#ver1[@]}; i<${#ver2[@]}; i++)) | ||
| + | do | ||
| + | ver1[i]=0 | ||
| + | done | ||
| + | for ((i=0; i<${#ver1[@]}; i++)) | ||
| + | do | ||
| + | if [[ -z ${ver2[i]} ]] | ||
| + | then | ||
| + | # fill empty fields in ver2 with zeros | ||
| + | ver2[i]=0 | ||
| + | fi | ||
| + | if ((10#${ver1[i]} > 10#${ver2[i]})) | ||
| + | then | ||
| + | return 1 | ||
| + | fi | ||
| + | if ((10#${ver1[i]} < 10#${ver2[i]})) | ||
| + | then | ||
| + | return 2 | ||
| + | fi | ||
| + | done | ||
| + | return 0 | ||
| + | } | ||
| + | |||
| + | |||
| + | # | ||
| + | # Copy a file or directory if exists | ||
| + | # Arguments: | ||
| + | # $1: file/directory to copy | ||
| + | # $2: destination directory (create if necessary) | ||
| + | # | ||
| + | copy_if_exists(){ | ||
| + | local from="$1" | ||
| + | local to="$2" | ||
| + | |||
| + | if [ -e "${from}" ] ; then | ||
| + | [ ! -d "${to}" ] && mkdir -p "${to}" | ||
| + | cp -a "${from}" "${to}" | ||
| + | fi | ||
| + | } | ||
| + | |||
| + | |||
| + | # | ||
| + | # Check KerOS/SPN version and stop if: | ||
| + | # - KerOS < 4.0 | ||
| + | # - SPN < 2.1 | ||
| + | # | ||
| + | check_fw_version(){ | ||
| # check if /etc/version exists | # check if /etc/version exists | ||
| if [ -a /etc/version ]; then | if [ -a /etc/version ]; then | ||
| - | FW_VERSION=$(cat /etc/version) | + | FW_VERSION=$(cat /etc/version) |
| - | # Check if the firmware version is superior or equal to 4. If not, stop the script. | + | # Detect type of firmware: KerOS or SPN |
| - | if [ $(cut -c 1 <<< $FW_VERSION) -lt 4 ]; then | + | if [[ "${FW_VERSION}" == *-spn-* ]]; then |
| - | echo "Your firmware version ($FW_VERSION) is inferior to v4. Please use the adequate script." | + | # Check if SPN version is not inferior to 2.1, stop if it is the case |
| - | exit | + | compare_versions ${FW_VERSION:0:3} "2.1" |
| - | fi | + | if [ $? -eq 2 ]; then |
| + | echo "Your SPN firmware version ($FW_VERSION) is inferior to v2.1. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt | ||
| + | exit | ||
| + | fi | ||
| + | else | ||
| + | # Check if KerOS version is not inferior to 4.0, stop if it is the case | ||
| + | compare_versions ${FW_VERSION:0:3} "4.0" | ||
| + | if [ $? -eq 2 ]; then | ||
| + | echo "Your KerOS firmware version ($FW_VERSION) is inferior to v4. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt | ||
| + | exit | ||
| + | fi | ||
| + | fi | ||
| else | else | ||
| - | echo "could not find /etc/version" | + | echo "Could not find /etc/version" >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt |
| - | exit | + | exit |
| fi | fi | ||
| } | } | ||
| + | |||
| # | # | ||
| # Check platform we are running on | # Check platform we are running on | ||
| # | # | ||
| - | get_platform() { | + | detect_platform() { |
| - | platform="$(fslotpreg getplatform)" | + | PLATFORM="$(fslotpreg getplatform)" |
| if [ $? -ne 0 ]; | if [ $? -ne 0 ]; | ||
| then | then | ||
| - | platform="NOT_SUPPORTED" | + | PLATFORM="NOT_SUPPORTED" |
| fi | fi | ||
| - | echo ${platform} | + | echo "Platform detected: ${PLATFORM}" |
| } | } | ||
| - | get_logs() | + | # |
| - | { | + | # Get product serial number information: SERIAL and SHORT_MAC |
| - | NAME="Logs_${serial}_${shortmac}_$(date "+%Y-%m-%d_%H-%M-%S")" | + | # |
| - | tmpdir="/tmp/${NAME}" | + | fill_serials() { |
| + | local reg=$(fslotpreg read GP1) | ||
| + | if [ $? -eq 0 ]; then | ||
| + | SERIAL=$(printf "%X\n" $reg) | ||
| + | else | ||
| + | SERIAL="unknown" | ||
| + | fi | ||
| + | SHORT_MAC=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}') | ||
| + | } | ||
| - | mkdir -p ${tmpdir}/ | ||
| - | # get logs | + | # |
| - | mkdir -p ${tmpdir}/var/log | + | # Detect the type of environment where the script is launched. |
| - | cp -a /var/log/* ${tmpdir}/var/log/ | + | # Possible environments: |
| + | # - udev | ||
| + | # - shell | ||
| + | # | ||
| + | detect_env() { | ||
| + | # Detection is only based on the presence or not of the variable DEVNAME | ||
| + | if [ ! -z ${DEVNAME} ] ; then | ||
| + | SCRIPT_ENV=udev | ||
| + | exec > /dev/console 2>&1 # Output the script execution in debug console | ||
| + | else | ||
| + | SCRIPT_ENV=shell | ||
| + | LEDS_MANAGEMENT=false | ||
| + | OUTPUT_DIR=$(pwd) | ||
| + | fi | ||
| + | echo "Script environment: ${SCRIPT_ENV}" | ||
| + | } | ||
| - | # get tmp files | + | |
| - | mkdir -p ${tmpdir}/tmp | + | # |
| - | cp -a /tmp/sys_startup_status.json ${tmpdir}/tmp/ | + | # Detect type of leds depending of platform/firmware version |
| - | if [ ${platform} == "IBTS" ]; then | + | # |
| - | cp -a /tmp/calib_loraloc.json ${tmpdir}/tmp/ | + | detect_leds() { |
| - | elif [ ${platform} == "IFEMTO" ]; then | + | if [ -e "${SYSLEDPATH}" ]; then |
| - | cp -a /tmp/calib_rf.json ${tmpdir}/tmp/ | + | LED_PATH="${SYSLEDPATH}" |
| + | elif [ -e "${SYSLEDPATH_4_0_IBTS}" ]; then | ||
| + | LED_PATH="${SYSLEDPATH_4_0_IBTS}" | ||
| + | elif [ -e "${SYSLEDPATH_4_0_IFEMTO}" ]; then | ||
| + | LED_PATH="${SYSLEDPATH_4_0_IFEMTO}" | ||
| fi | fi | ||
| + | } | ||
| - | # get network config | ||
| - | mkdir -p ${tmpdir}/etc/network | ||
| - | mkdir -p ${tmpdir}/etc/firewall.d | ||
| - | cp -a /etc/network/* ${tmpdir}/etc/network/ | ||
| - | cp -a /etc/firewall.d/* ${tmpdir}/etc/firewall.d/ | ||
| - | # CPF config | + | # |
| - | mkdir -p ${tmpdir}/etc/default | + | # Set default led behavior |
| - | mkdir -p ${tmpdir}/user/etc/lorad | + | # $1: trigger |
| - | mkdir -p ${tmpdir}/user/etc/lorafwd | + | # |
| - | cp -a /etc/default/lorad ${tmpdir}/etc/default/ | + | # if trigger = timer : $2 = delay_on, $3 = delay_off |
| - | cp -a /etc/default/lorafwd ${tmpdir}/etc/default/ | + | # |
| - | cp -a /user/etc/lorad/* ${tmpdir}/user/etc/lorad/ | + | set_led() { |
| - | cp -a /user/etc/lorafwd/* ${tmpdir}/user/etc/lorafwd/ | + | trigger="${1}" |
| + | on="$2" | ||
| + | off="$3" | ||
| - | # miscaellenous config | + | if [ -e "${LED_PATH}" ]; then |
| - | mkdir -p ${tmpdir}/etc/rcU.d | + | echo $trigger > "${LED_PATH}/trigger" |
| - | cp -a /etc/rcU.d/* ${tmpdir}/etc/rcU.d | + | case "$trigger" in |
| - | cp -a /user/.update ${tmpdir}/user/.update | + | "timer") |
| + | echo ${on:=500} > "${LED_PATH}/delay_on" | ||
| + | echo ${off:=500} > "${LED_PATH}/delay_off" | ||
| + | ;; | ||
| + | "none") | ||
| + | echo ${on:=0} > "${LED_PATH}/brightness" | ||
| + | ;; | ||
| + | *) | ||
| + | # Nothing to do | ||
| + | ;; | ||
| + | esac | ||
| + | fi | ||
| + | } | ||
| - | ( | + | get_nmea_frames() { |
| + | gps_tty="$1" | ||
| + | |||
| + | if ! [ -e "$gps_tty" ]; then | ||
| + | echo "$gps_tty doesn't exist" | ||
| + | return 1 | ||
| + | fi | ||
| + | |||
| + | if [ -n "$(lsof | grep $(readlink $gps_tty))" ]; then | ||
| + | echo "$gps_tty already used by $(lsof | grep $(readlink $gps_tty))" | ||
| + | return 2 | ||
| + | else | ||
| + | # Pseudo-term keep only FIRST frames so we keep last of these old frames and get few new frames | ||
| + | # It will take 3 seconds | ||
| + | timeout -t 3 cat $gps_tty > /tmp/nmea.log | ||
| + | cat /tmp/nmea.log | ||
| + | fi | ||
| + | } | ||
| + | |||
| + | # | ||
| + | # Execute some diagnosis commands (can take a long time) | ||
| + | # | ||
| + | execute_commands() { | ||
| + | echo "Executing diagnosis commands (can take a long time)" | ||
| + | |||
| + | echo -e "\tSystems commands" | ||
| + | ( | ||
| + | echo -e "\n\n\n************** fslotpreg getplatform *****************" | ||
| + | fslotpreg getplatform | ||
| + | echo -e "\n\n\n************** fslotpreg fslotpreg read GP1 *****************" | ||
| + | fslotpreg read GP1 | ||
| echo -e "\n\n\n************** date *****************" | echo -e "\n\n\n************** date *****************" | ||
| date | date | ||
| echo -e "\n\n\n************** uptime *****************" | echo -e "\n\n\n************** uptime *****************" | ||
| uptime | uptime | ||
| - | echo -e "\n\n\n************** cat /etc/version *****************" | + | echo -e "\n\n\n************** ps ww *****************" |
| - | cat /etc/version | + | ps ww |
| - | echo -e "\n\n\n************** ps w *****************" | + | echo -e "\n\n\n************** opkg status *****************" |
| - | ps w | + | opkg status |
| - | echo -e "\n\n\n************** opkg list-installed *****************" | + | echo -e "\n\n\n************** monit status *****************" |
| - | opkg list-installed | + | monit status |
| - | echo -e "\n\n\n************** opkg list *****************" | + | echo -e "\n\n\n************** mount *****************" |
| - | opkg list | + | mount |
| - | echo -e "\n\n\n************** monit status *****************" | + | |
| - | monit status | + | |
| - | echo -e "\n\n\n************** mount *****************" | + | |
| - | mount | + | |
| ) >> ${tmpdir}/cmd_results.txt | ) >> ${tmpdir}/cmd_results.txt | ||
| + | echo -e "\tNetwork commands" | ||
| ( | ( | ||
| echo -e "\n\n\n************** ifconfig -a *****************" | echo -e "\n\n\n************** ifconfig -a *****************" | ||
| Line 189: | Line 387: | ||
| gsmdiag.py ; cat /tmp/gsmdiag.txt | gsmdiag.py ; cat /tmp/gsmdiag.txt | ||
| echo -e "\n\n\n************** route *****************" | echo -e "\n\n\n************** route *****************" | ||
| - | route | + | route -n |
| echo -e "\n\n\n************** iptables -L *****************" | echo -e "\n\n\n************** iptables -L *****************" | ||
| iptables -L | iptables -L | ||
| Line 197: | Line 395: | ||
| connmanctl services | connmanctl services | ||
| - | if [ ${platform} == "IBTS" ]; then | + | if [ -e /dev/cdc-wdm1 ]; then |
| echo -e "\n\n\n************** qmicli get mode *****************" | echo -e "\n\n\n************** qmicli get mode *****************" | ||
| qmicli -d /dev/cdc-wdm1 --nas-get-system-selection-preference | qmicli -d /dev/cdc-wdm1 --nas-get-system-selection-preference | ||
| fi | fi | ||
| ) >> ${tmpdir}/network_result.txt | ) >> ${tmpdir}/network_result.txt | ||
| - | + | ||
| - | # get the 25 last NMEA frames; it will take 10s + 25s | + | get_nmea_frames /dev/nmea3 > ${tmpdir}/nmea.log |
| - | if [ ${platform} == "IBTS" ]; then | + | [ $? -ne 0 ] && get_nmea_frames /dev/nmea2 >> ${tmpdir}/nmea.log |
| - | + | ||
| - | if [ -n "$(lsof | grep $(readlink /dev/nmea2))" ]; then | + | # Overlay modifications |
| - | echo " /dev/nmea2 already used by $(lsof | grep $(readlink /dev/nmea2))" > ${tmpdir}/nmea2.log; | + | echo -e "\tOverlay commands" |
| - | + | overlay diff -l /.rootfs.ro -u /user/.rootfs_upper/ >> ${tmpdir}/overlay_status.txt << EOF | |
| - | else | + | yes |
| - | timeout -t 10 cat /dev/nmea2 > /dev/null | + | EOF |
| - | timeout -t 40 head -n 50 /dev/nmea2 > ${tmpdir}/nmea2.log | + | |
| - | fi | + | echo "End of diagnosis commands" |
| + | } | ||
| + | |||
| + | |||
| + | # | ||
| + | # Get all the logs, configuration files and commands and save it | ||
| + | # in the resquested output directory. | ||
| + | # Arguments: | ||
| + | # $1: output directory | ||
| + | # | ||
| + | get_logs() | ||
| + | { | ||
| + | local output_dir="$1" | ||
| + | local name="Logs_${SERIAL}_${SHORT_MAC}_$(date "+%Y%m%d-%H%M%S")" | ||
| + | local tmpdir="/user/${name}" | ||
| + | |||
| + | # tmp files | ||
| + | echo "Extracting tmp files" | ||
| + | copy_if_exists /tmp/sys_startup_status.json ${tmpdir}/tmp/ | ||
| + | copy_if_exists /tmp/board_info.json ${tmpdir}/tmp/ | ||
| + | copy_if_exists /tmp/calib_loraloc.json ${tmpdir}/tmp/ | ||
| + | copy_if_exists /tmp/calib_rf.json ${tmpdir}/tmp/ | ||
| + | |||
| + | # network configuration | ||
| + | echo "Extracting network configuration" | ||
| + | copy_if_exists /etc/network/ ${tmpdir}/etc/ | ||
| + | copy_if_exists /etc/firewall.d/ ${tmpdir}/etc/ | ||
| + | |||
| + | # CPF (lorad + lorafwd) configuration | ||
| + | echo "Extracting CPF configuration (if any)" | ||
| + | for lorad_dir in /user/etc/lorad /etc/lorad | ||
| + | do | ||
| + | if [ -d ${lorad_dir} ] ; then | ||
| + | echo -e "\tExtracting lorad configuration" | ||
| + | copy_if_exists /etc/default/lorad ${tmpdir}/etc/default/ | ||
| + | copy_if_exists ${lorad_dir} ${tmpdir}${lorad_dir%/*} | ||
| + | fi | ||
| + | done | ||
| + | for lorafwd_dir in /user/etc/lorafwd /etc/lorafwd | ||
| + | do | ||
| + | if [ -d ${lorafwd_dir} ] ; then | ||
| + | echo -e "\tExtracting lorafwd configuration" | ||
| + | copy_if_exists /etc/default/lorafwd ${tmpdir}/etc/default/ | ||
| + | copy_if_exists ${lorafwd_dir} ${tmpdir}${lorafwd_dir%/*} | ||
| + | fi | ||
| + | done | ||
| + | |||
| + | # WMC configuration and logs | ||
| + | echo "Extracting WMC configuration and logs (if any)" | ||
| + | if [ -d /user/bscc ] ; then | ||
| + | echo -e "\tExtracting BSCC configuration and logs" | ||
| + | copy_if_exists /user/bscc/var.xml ${tmpdir}/user/bscc/ | ||
| + | copy_if_exists /user/bscc/traces/ ${tmpdir}/user/bscc/ | ||
| + | fi | ||
| + | if [ -d /user/snmp ] ; then | ||
| + | echo -e "\tExtracting SNMP configuration and logs" | ||
| + | copy_if_exists /user/snmp/snmpd.conf ${tmpdir}/user/snmp/ | ||
| + | copy_if_exists /user/snmp/traces/ ${tmpdir}/user/snmp/ | ||
| fi | fi | ||
| - | # create an archive with all the files and command results | + | # miscellaneous configuration and logs |
| - | # the archive is created on the USB key | + | echo "Extracting miscellaneous configuration and logs" |
| - | tar -C /tmp -czf ${NAME}.tar.gz ${NAME} | + | copy_if_exists /etc/rcU.d/ ${tmpdir}/etc/ |
| - | echo "Archive ${NAME}.tar.gz created" | + | copy_if_exists /.update/update.log ${tmpdir}/ |
| + | copy_if_exists /user/.update ${tmpdir}/user/ | ||
| + | copy_if_exists /user/.upgrade.log ${tmpdir}/user/ | ||
| + | copy_if_exists /user/.sysupgrade.log ${tmpdir}/user/ | ||
| + | copy_if_exists /etc/version ${tmpdir}/etc/ | ||
| + | copy_if_exists /etc/sysupgrade.conf ${tmpdir}/etc/ | ||
| + | copy_if_exists /etc/sysupgrade.d/ ${tmpdir}/etc/ | ||
| + | copy_if_exists /etc/.squash_layer_version ${tmpdir}/etc/ | ||
| + | echo ${SCRIPT_VERSION} > ${tmpdir}/script_version | ||
| + | |||
| + | if ${EXEC_COMMANDS} ; then | ||
| + | execute_commands | ||
| + | fi | ||
| + | |||
| + | # Finish by system logs to prevent space errors | ||
| + | echo "Extracting system logs" | ||
| + | copy_if_exists /var/log/ ${tmpdir}/var/ | ||
| + | |||
| + | echo "Saving all data in ${output_dir}/${name}.tar.gz" | ||
| + | tar -C /user -czf ${output_dir}/${name}.tar.gz ${name} | ||
| + | echo "Done" | ||
| sync | sync | ||
| Line 229: | Line 504: | ||
| ##################### | ##################### | ||
| - | check_version | + | # Detect the environment of execution of the script |
| + | detect_env | ||
| - | platform=$(get_platform) | + | # Manage scripts arguments |
| + | while [[ $# -gt 0 ]] | ||
| + | do | ||
| + | key="$1" | ||
| - | reg=$(fslotpreg read GP1) | + | case $key in |
| - | if [ $? -eq 0 ]; then | + | -l|--ignore-leds) |
| - | serial=$(printf "%X\n" $reg) | + | LEDS_MANAGEMENT=false |
| - | else | + | shift |
| - | serial="unknown" | + | ;; |
| + | -o|--output-dir) | ||
| + | OUTPUT_DIR="$2" | ||
| + | shift 2 | ||
| + | ;; | ||
| + | -c|--disable-commands) | ||
| + | EXEC_COMMANDS=false | ||
| + | shift | ||
| + | ;; | ||
| + | -h|--help) | ||
| + | show_help | ||
| + | shift | ||
| + | exit 1 | ||
| + | ;; | ||
| + | -v|--version) | ||
| + | show_version | ||
| + | shift | ||
| + | exit 1 | ||
| + | ;; | ||
| + | *) | ||
| + | shift | ||
| + | ;; | ||
| + | esac | ||
| + | done | ||
| + | |||
| + | # Check if firmware version is compatible | ||
| + | check_fw_version | ||
| + | |||
| + | # Detect type of platform | ||
| + | detect_platform | ||
| + | |||
| + | # Fill SERIAL and SHORT_MAC variables | ||
| + | fill_serials | ||
| + | |||
| + | # Set led On | ||
| + | if ${LEDS_MANAGEMENT} ; then | ||
| + | detect_leds | ||
| + | set_led none 1 | ||
| fi | fi | ||
| - | shortmac=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}') | ||
| # Get logs | # Get logs | ||
| - | get_logs | + | get_logs ${OUTPUT_DIR} |
| + | |||
| + | # End of script, manage user's notifications | ||
| + | if ${LEDS_MANAGEMENT} ; then | ||
| + | # Prepare reboot after USB unplug | ||
| + | cat > /tmp/usbkey_remove << EOF | ||
| + | # Switch off Led | ||
| + | echo "none" > ${LED_PATH}/trigger | ||
| + | EOF | ||
| + | |||
| + | # All is right, make Led blinking fast to notify User | ||
| + | set_led timer 100 100 | ||
| + | fi | ||
| </code> | </code> | ||
| + | |||
| + | |||
| ++++ | ++++ | ||
| Line 250: | Line 579: | ||
| To gather logs from a USB without typing any command: | To gather logs from a USB without typing any command: | ||
| - | * Download the following script and copy it on a USB key. | + | * Download the following script and copy it on a USB flash drive. |
| - | * Create a ''usbkey.txt'' file on the USB key as described [[wiki:sw_updates#specific_files|here]]. | + | * Create a ''usbkey.txt'' file on the USB flash drive as described [[wiki:keros_custo:sw_updates#specific_files|here]]. |
| - | * Plug the USB key on the CPU board. | + | * Plug the USB flash drive into the USB port of the gateway. |
| * Wait for the yellow debug led fast blinking. //The execution of the script takes about 1-2 minutes.// | * Wait for the yellow debug led fast blinking. //The execution of the script takes about 1-2 minutes.// | ||
| - | * Unplug the USB key. | + | * Unplug the USB flash drive. |
| - | * An archive with all the required logs has been created on the USB key. | + | * An archive with all the required logs has been created on the USB flash drive. |
| <code bash usb.autorun> | <code bash usb.autorun> | ||
| #!/bin/sh | #!/bin/sh | ||
| - | USBDIR="`dirname $0`" | + | # Version of the script |
| + | SCRIPT_VERSION=1.4 | ||
| - | SYSLEDPATH_IBTS="/sys/class/leds/yellow:debug/" | + | # Output directory for the result of the script |
| - | SYSLEDPATH_IFEMTO="/sys/class/leds/led1:red:power/" | + | OUTPUT_DIR="`dirname $0`" |
| + | # Execution's environment of the script | ||
| + | SCRIPT_ENV=unknown | ||
| - | check_version(){ | + | # PLATFORM: IBTS, IFEMTO, ISTATION, IFEVO |
| - | # check if /etc/version exists | + | PLATFORM=unknown |
| - | if [ -a /etc/version ]; then | + | # Serial number of the product: 39020003 |
| - | FW_VERSION=$(cat /etc/version) | + | SERIAL=unknown |
| + | # ETH0 MAC: 70:76:ff:01:77:71 => 7771 | ||
| + | SHORT_MAC=unknown | ||
| - | # Check if the firmware version is superior or equal to 4. If not, stop the script. | + | # LED paths |
| - | if [ $(cut -c 1 <<< $FW_VERSION) -lt 4 ]; then | + | SYSLEDPATH_4_0_IBTS="/sys/class/leds/yellow:debug" |
| - | echo "Your firmware version ($FW_VERSION) is inferior to v4. Please use the adequate script." >> ${USBDIR}/WRONG_FIRMWARE_VERSION.txt | + | SYSLEDPATH_4_0_IFEMTO="/sys/class/leds/led1:red:power" |
| - | exit | + | SYSLEDPATH="/sys/class/leds/status" |
| - | fi | + | LED_PATH="/do/not/exist" |
| - | else | + | |
| - | echo "Could not find /etc/version" >> ${USBDIR}/WRONG_FIRMWARE_VERSION.txt | + | # Script arguments |
| - | exit | + | LEDS_MANAGEMENT=true |
| - | fi | + | EXEC_COMMANDS=true |
| + | |||
| + | # | ||
| + | # Print the version of the script | ||
| + | # | ||
| + | show_version() { | ||
| + | echo "Script version: ${SCRIPT_VERSION}" | ||
| + | } | ||
| + | |||
| + | |||
| + | # | ||
| + | # Show help | ||
| + | # | ||
| + | show_help() | ||
| + | { | ||
| + | show_version | ||
| + | echo -e "Usage: $0 [-h|--help] [-l|--ignore-leds] [-o|--output-dir <directory>] | ||
| + | \t-h|--help: this help | ||
| + | \t-v|--version: version of the script | ||
| + | \t-l|--ignore-leds: this script does not manage the leds (default: autodetection) | ||
| + | \t-o|--output-dir <directory>: force the output <directory> to use for extraction (default: current directory) | ||
| + | \t-c|--disable-commands: disable diagnosis commands (default: executed) | ||
| + | \t-m|--minimal: get minimal logs (to lower generated file size) | ||
| + | " | ||
| + | } | ||
| + | |||
| + | # | ||
| + | # Compare 2 versions to know if inferior or superior | ||
| + | # Res: 0: $1 = $2 | ||
| + | # 1: $1 > $2 | ||
| + | # 2: $1 < $2 | ||
| + | # | ||
| + | compare_versions () { | ||
| + | if [[ $1 == $2 ]] | ||
| + | then | ||
| + | return 0 | ||
| + | fi | ||
| + | local IFS=. | ||
| + | local i ver1=($1) ver2=($2) | ||
| + | # fill empty fields in ver1 with zeros | ||
| + | for ((i=${#ver1[@]}; i<${#ver2[@]}; i++)); do | ||
| + | ver1[i]=0 | ||
| + | done | ||
| + | for ((i=0; i<${#ver1[@]}; i++)); do | ||
| + | if [[ -z ${ver2[i]} ]] | ||
| + | then | ||
| + | # fill empty fields in ver2 with zeros | ||
| + | ver2[i]=0 | ||
| + | fi | ||
| + | if ((10#${ver1[i]} > 10#${ver2[i]})) | ||
| + | then | ||
| + | return 1 | ||
| + | fi | ||
| + | if ((10#${ver1[i]} < 10#${ver2[i]})) | ||
| + | then | ||
| + | return 2 | ||
| + | fi | ||
| + | done | ||
| + | return 0 | ||
| + | } | ||
| + | |||
| + | |||
| + | # | ||
| + | # Copy a file or directory if exists | ||
| + | # Arguments: | ||
| + | # $1: file/directory to copy | ||
| + | # $2: destination directory (create if necessary) | ||
| + | # | ||
| + | copy_if_exists(){ | ||
| + | local from="$1" | ||
| + | local to="$2" | ||
| + | |||
| + | if [ -e "${from}" ] ; then | ||
| + | [ ! -d "${to}" ] && mkdir -p "${to}" | ||
| + | cp -a "${from}" "${to}" | ||
| + | fi | ||
| + | } | ||
| + | |||
| + | |||
| + | # | ||
| + | # Check KerOS/SPN version and stop if: | ||
| + | # - KerOS < 4.0 | ||
| + | # - SPN < 2.1 | ||
| + | # | ||
| + | check_fw_version(){ | ||
| + | # check if /etc/version exists | ||
| + | if [ -a /etc/version ]; then | ||
| + | FW_VERSION=$(cat /etc/version) | ||
| + | |||
| + | # Detect type of firmware: KerOS or SPN | ||
| + | if [[ "${FW_VERSION}" == *-spn-* ]]; then | ||
| + | # Check if SPN version is not inferior to 2.1, stop if it is the case | ||
| + | compare_versions ${FW_VERSION:0:3} "2.1" | ||
| + | if [ $? -eq 2 ]; then | ||
| + | echo "Your SPN firmware version ($FW_VERSION) is inferior to v2.1. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt | ||
| + | exit | ||
| + | fi | ||
| + | else | ||
| + | # Check if KerOS version is not inferior to 4.0, stop if it is the case | ||
| + | compare_versions ${FW_VERSION:0:3} "4.0" | ||
| + | if [ $? -eq 2 ]; then | ||
| + | echo "Your KerOS firmware version ($FW_VERSION) is inferior to v4. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt | ||
| + | exit | ||
| + | fi | ||
| + | fi | ||
| + | else | ||
| + | echo "Could not find /etc/version" >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt | ||
| + | exit | ||
| + | fi | ||
| } | } | ||
| Line 286: | Line 729: | ||
| # Check platform we are running on | # Check platform we are running on | ||
| # | # | ||
| - | get_platform() { | + | detect_platform() { |
| - | platform="$(fslotpreg getplatform)" | + | PLATFORM="$(fslotpreg getplatform)" |
| - | if [ $? -ne 0 ]; | + | if [ $? -ne 0 ]; |
| - | then | + | then |
| - | platform="NOT_SUPPORTED" | + | PLATFORM="NOT_SUPPORTED" |
| - | fi | + | fi |
| - | echo ${platform} | + | echo "Platform detected: ${PLATFORM}" |
| + | } | ||
| + | |||
| + | |||
| + | # | ||
| + | # Get product serial number information: SERIAL and SHORT_MAC | ||
| + | # | ||
| + | fill_serials() { | ||
| + | local reg=$(fslotpreg read GP1) | ||
| + | if [ $? -eq 0 ]; then | ||
| + | SERIAL=$(printf "%X\n" $reg) | ||
| + | else | ||
| + | SERIAL="unknown" | ||
| + | fi | ||
| + | if [ -f "/sys/class/net/eth0/address" ]; then | ||
| + | SHORT_MAC=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}') | ||
| + | elif [ -f "/sys/class/net/usb0/address" ]; then | ||
| + | SHORT_MAC=$(cat /sys/class/net/usb0/address | awk -F: '{print $5$6}') | ||
| + | else | ||
| + | SHORT_MAC="XXXX" | ||
| + | fi | ||
| + | } | ||
| + | |||
| + | |||
| + | # | ||
| + | # Detect the type of environment where the script is launched. | ||
| + | # Possible environments: | ||
| + | # - udev | ||
| + | # - shell | ||
| + | # | ||
| + | detect_env() { | ||
| + | # Detection is only based on the presence or not of the variable DEVNAME | ||
| + | if [ ! -z ${DEVNAME} ] ; then | ||
| + | SCRIPT_ENV=udev | ||
| + | exec > /dev/console 2>&1 # Output the script execution in debug console | ||
| + | else | ||
| + | SCRIPT_ENV=shell | ||
| + | LEDS_MANAGEMENT=false | ||
| + | OUTPUT_DIR=$(pwd) | ||
| + | fi | ||
| + | echo "Script environment: ${SCRIPT_ENV}" | ||
| + | } | ||
| + | |||
| + | |||
| + | # | ||
| + | # Detect type of leds depending of platform/firmware version | ||
| + | # | ||
| + | detect_leds() { | ||
| + | if [ -e "${SYSLEDPATH}" ]; then | ||
| + | LED_PATH="${SYSLEDPATH}" | ||
| + | elif [ -e "${SYSLEDPATH_4_0_IBTS}" ]; then | ||
| + | LED_PATH="${SYSLEDPATH_4_0_IBTS}" | ||
| + | elif [ -e "${SYSLEDPATH_4_0_IFEMTO}" ]; then | ||
| + | LED_PATH="${SYSLEDPATH_4_0_IFEMTO}" | ||
| + | fi | ||
| } | } | ||
| Line 303: | Line 800: | ||
| # | # | ||
| set_led() { | set_led() { | ||
| - | trigger="${1}" | + | trigger="${1}" |
| - | on="$2" | + | on="$2" |
| - | off="$3" | + | off="$3" |
| - | if [ -d "${SYSLEDPATH}" ]; then | + | if [ -e "${LED_PATH}" ]; then |
| - | echo $trigger > "${SYSLEDPATH}/trigger" | + | echo $trigger > "${LED_PATH}/trigger" |
| - | case "$trigger" in | + | case "$trigger" in |
| - | "timer") | + | "timer") |
| - | echo ${on:=500} > "${SYSLEDPATH}/delay_on" | + | echo ${on:=500} > "${LED_PATH}/delay_on" |
| - | echo ${off:=500} > "${SYSLEDPATH}/delay_off" | + | echo ${off:=500} > "${LED_PATH}/delay_off" |
| - | ;; | + | ;; |
| - | "none") | + | "none") |
| - | echo ${on:=0} > "${SYSLEDPATH}/brightness" | + | echo ${on:=0} > "${LED_PATH}/brightness" |
| - | ;; | + | ;; |
| - | *) | + | *) |
| - | # Nothing to do | + | # Nothing to do |
| - | ;; | + | ;; |
| - | esac | + | esac |
| - | fi | + | fi |
| } | } | ||
| + | get_nmea_frames() { | ||
| + | gps_tty="$1" | ||
| + | if ! [ -e "$gps_tty" ]; then | ||
| + | echo "$gps_tty doesn't exist" | ||
| + | return 1 | ||
| + | fi | ||
| + | |||
| + | if [ -n "$(lsof | grep $(readlink $gps_tty))" ]; then | ||
| + | echo "$gps_tty already used by $(lsof | grep $(readlink $gps_tty))" | ||
| + | return 2 | ||
| + | else | ||
| + | # Pseudo-term keep only FIRST frames so we keep last of these old frames and get few new frames | ||
| + | # It will take 3 seconds | ||
| + | timeout -t 3 cat $gps_tty > /tmp/nmea.log | ||
| + | cat /tmp/nmea.log | ||
| + | fi | ||
| + | } | ||
| + | |||
| + | # | ||
| + | # Execute some diagnosis commands (can take a long time) | ||
| + | # | ||
| + | execute_commands() { | ||
| + | echo "Executing diagnosis commands (can take a long time)" | ||
| + | |||
| + | echo -e "\tSystems commands" | ||
| + | ( | ||
| + | echo -e "\n\n\n************** fslotpreg getplatform *****************" | ||
| + | fslotpreg getplatform | ||
| + | echo -e "\n\n\n************** fslotpreg fslotpreg read GP1 *****************" | ||
| + | fslotpreg read GP1 | ||
| + | echo -e "\n\n\n************** date *****************" | ||
| + | date | ||
| + | echo -e "\n\n\n************** uptime *****************" | ||
| + | uptime | ||
| + | echo -e "\n\n\n************** ps ww *****************" | ||
| + | ps ww | ||
| + | echo -e "\n\n\n************** opkg status *****************" | ||
| + | opkg status | ||
| + | echo -e "\n\n\n************** monit status *****************" | ||
| + | monit status | ||
| + | echo -e "\n\n\n************** mount *****************" | ||
| + | mount | ||
| + | echo -e "\n\n\n************** OpenVPN Certificate expiration date *****************" | ||
| + | compare_versions ${FW_VERSION:0:3} "5.0" | ||
| + | ret=$? | ||
| + | if [ $ret -eq 1 ]; then | ||
| + | if [ -f "/etc/openvpn/bscc.p12" ]; then | ||
| + | openssl pkcs12 -info -in /etc/openvpn/bscc.p12 -nokeys -password file:/etc/openvpn/bscc.password| openssl x509 -noout -enddate | ||
| + | fi | ||
| + | elif [ $ret -eq 2 ]; then | ||
| + | grep -iA3 "Validity" /.update/update.log | tail -3 | grep -i "not after" | sed 's/^[ \t]*//' | ||
| + | fi | ||
| + | echo -e "\n\n\n************** Disk info (df -h) *****************" | ||
| + | df -h | ||
| + | echo -e "\n\n\n************** Log Size (du -sh /var/log) *****************" | ||
| + | du -sh /var/log | ||
| + | ) >> ${tmpdir}/cmd_results.txt | ||
| + | |||
| + | echo -e "\tNetwork commands" | ||
| + | ( | ||
| + | echo -e "\n\n\n************** ifconfig -a *****************" | ||
| + | ifconfig -a | ||
| + | echo -e "\n\n\n************** gsmdiag.py *****************" | ||
| + | gsmdiag.py ; cat /tmp/gsmdiag.txt | ||
| + | echo -e "\n\n\n************** route *****************" | ||
| + | route -n | ||
| + | echo -e "\n\n\n************** iptables -L *****************" | ||
| + | iptables -L | ||
| + | echo -e "\n\n\n************** connmanctl technologies *****************" | ||
| + | connmanctl technologies | ||
| + | echo -e "\n\n\n************** services *****************" | ||
| + | connmanctl services | ||
| + | |||
| + | if [ -e /dev/cdc-wdm1 ]; then | ||
| + | echo -e "\n\n\n************** qmicli get mode *****************" | ||
| + | qmicli -d /dev/cdc-wdm1 --nas-get-system-selection-preference | ||
| + | fi | ||
| + | ) >> ${tmpdir}/network_result.txt | ||
| + | |||
| + | get_nmea_frames /dev/nmea3 > ${tmpdir}/nmea.log | ||
| + | [ $? -ne 0 ] && get_nmea_frames /dev/nmea2 >> ${tmpdir}/nmea.log | ||
| + | |||
| + | # Overlay modifications | ||
| + | echo -e "\tOverlay commands disabled (KEROS-1907)" | ||
| + | # overlay diff -l /.rootfs.ro -u /user/.rootfs_upper/ >> ${tmpdir}/overlay_status.txt << EOF | ||
| + | #yes | ||
| + | #EOF | ||
| + | |||
| + | echo "End of diagnosis commands" | ||
| + | } | ||
| + | |||
| + | |||
| + | # | ||
| + | # Get all the logs, configuration files and commands and save it | ||
| + | # in the resquested output directory. | ||
| + | # Arguments: | ||
| + | # $1: output directory | ||
| + | # | ||
| get_logs() | get_logs() | ||
| { | { | ||
| - | NAME="Logs_${serial}_${shortmac}_$(date "+%Y-%m-%d_%H-%M-%S")" | + | local output_dir="$1" |
| - | tmpdir="/tmp/${NAME}" | + | local option="$2" |
| + | local name="Logs_${SERIAL}_${SHORT_MAC}_$(date "+%Y%m%d-%H%M%S")" | ||
| + | if [ "${option}" = "minimal" ]; then | ||
| + | name="${name}_minimal" | ||
| + | fi | ||
| + | local tmpdir="/user/${name}" | ||
| - | mkdir -p ${tmpdir}/ | + | # tmp files |
| + | echo "Extracting tmp files" | ||
| + | copy_if_exists /tmp/sys_startup_status.json ${tmpdir}/tmp/ | ||
| + | copy_if_exists /tmp/board_info.json ${tmpdir}/tmp/ | ||
| + | copy_if_exists /tmp/calib_loraloc.json ${tmpdir}/tmp/ | ||
| + | copy_if_exists /tmp/calib_rf.json ${tmpdir}/tmp/ | ||
| - | # get logs | + | # network configuration |
| - | mkdir -p ${tmpdir}/var/log | + | echo "Extracting network configuration" |
| - | cp -a /var/log/* ${tmpdir}/var/log/ | + | if [ "${option}" = "minimal" ]; then |
| + | copy_if_exists /etc/network/connman/ ${tmpdir}/etc/network/ | ||
| + | copy_if_exists /etc/network/ofono/ ${tmpdir}/etc/network/ | ||
| + | copy_if_exists /etc/network/openvpn/ ${tmpdir}/etc/network/ | ||
| + | cp /etc/network/*.conf ${tmpdir}/etc/network/ | ||
| + | else | ||
| + | copy_if_exists /etc/network/ ${tmpdir}/etc/ | ||
| + | fi | ||
| + | copy_if_exists /etc/firewall.d/ ${tmpdir}/etc/ | ||
| - | # get tmp files | + | # init configuration |
| - | mkdir -p ${tmpdir}/tmp | + | if [ "${option}" != "minimal" ]; then |
| - | cp -a /tmp/sys_startup_status.json ${tmpdir}/tmp/ | + | echo "Extracting init.d and rc*.d files" |
| - | if [ ${platform} == "IBTS" ]; then | + | copy_if_exists /etc/init.d/ ${tmpdir}/etc/ |
| - | cp -a /tmp/calib_loraloc.json ${tmpdir}/tmp/ | + | for filename in /etc/rc*.d; do |
| - | elif [ ${platform} == "IFEMTO" ]; then | + | copy_if_exists ${filename} ${tmpdir}/etc/ |
| - | cp -a /tmp/calib_rf.json ${tmpdir}/tmp/ | + | done |
| - | fi | + | fi |
| - | # get network config | + | # CPF (lorad + lorafwd) configuration |
| - | mkdir -p ${tmpdir}/etc/network | + | echo "Extracting CPF configuration (if any)" |
| - | mkdir -p ${tmpdir}/etc/firewall.d | + | if [ "${option}" = "minimal" ]; then |
| - | cp -a /etc/network/* ${tmpdir}/etc/network/ | + | copy_if_exists /user/etc/lorad/lorad.json ${tmpdir}/user/etc/lorad/ |
| - | cp -a /etc/firewall.d/* ${tmpdir}/etc/firewall.d/ | + | copy_if_exists /etc/lorad/lorad.json ${tmpdir}/etc/lorad/ |
| + | else | ||
| + | for lorad_dir in /user/etc/lorad /etc/lorad; do | ||
| + | if [ -d ${lorad_dir} ] ; then | ||
| + | echo -e "\tExtracting lorad configuration" | ||
| + | copy_if_exists ${lorad_dir} ${tmpdir}${lorad_dir%/*} | ||
| + | fi | ||
| + | done | ||
| + | fi | ||
| + | copy_if_exists /etc/default/lorad ${tmpdir}/etc/default/ | ||
| - | # CPF config | + | echo -e "\tExtracting lorafwd configuration" |
| - | mkdir -p ${tmpdir}/etc/default | + | if [ "${option}" = "minimal" ]; then |
| - | mkdir -p ${tmpdir}/user/etc/lorad | + | for filename in /etc/lorafwd/*.toml; do |
| - | mkdir -p ${tmpdir}/user/etc/lorafwd | + | copy_if_exists ${filename} ${tmpdir}/etc/lorafwd/ |
| - | cp -a /etc/default/lorad ${tmpdir}/etc/default/ | + | done |
| - | cp -a /etc/default/lorafwd ${tmpdir}/etc/default/ | + | else |
| - | cp -a /user/etc/lorad/* ${tmpdir}/user/etc/lorad/ | + | for lorafwd_dir in /user/etc/lorafwd /etc/lorafwd; do |
| - | cp -a /user/etc/lorafwd/* ${tmpdir}/user/etc/lorafwd/ | + | if [ -d ${lorafwd_dir} ] ; then |
| + | copy_if_exists ${lorafwd_dir} ${tmpdir}${lorafwd_dir%/*} | ||
| + | fi | ||
| + | done | ||
| + | fi | ||
| + | copy_if_exists /etc/lorafwd.toml ${tmpdir}/etc/ | ||
| + | copy_if_exists /etc/default/lorafwd ${tmpdir}/etc/default/ | ||
| - | # miscaellenous config | + | # WMC configuration and logs |
| - | mkdir -p ${tmpdir}/etc/rcU.d | + | echo "Extracting WMC configuration and logs (if any)" |
| - | cp -a /etc/rcU.d/* ${tmpdir}/etc/rcU.d | + | if [ -d /user/bscc ] ; then |
| - | cp -a /user/.update ${tmpdir}/user/.update | + | echo -e "\tExtracting BSCC configuration and logs" |
| + | copy_if_exists /user/bscc/var.xml ${tmpdir}/user/bscc/ | ||
| + | copy_if_exists /user/bscc/traces/ ${tmpdir}/user/bscc/ | ||
| + | fi | ||
| + | if [ -d /user/snmp ] ; then | ||
| + | echo -e "\tExtracting SNMP configuration and logs" | ||
| + | copy_if_exists /user/snmp/snmpd.conf ${tmpdir}/user/snmp/ | ||
| + | copy_if_exists /user/snmp/traces/ ${tmpdir}/user/snmp/ | ||
| + | fi | ||
| + | copy_if_exists /etc/default/bscc ${tmpdir}/etc/default/ | ||
| + | copy_if_exists /etc/default/lora-snmp ${tmpdir}/etc/default/ | ||
| + | copy_if_exists /etc/snmp/snmpd.conf ${tmpdir}/etc/snmp/ | ||
| - | ( | + | # basic station configuration and logs |
| - | echo -e "\n\n\n************** date *****************" | + | echo "Extracting basic station configuration and logs (if any)" |
| - | date | + | if [ -d /user/basic_station ] ; then |
| - | echo -e "\n\n\n************** uptime *****************" | + | echo -e "\tExtracting Basic Station configuration and logs" |
| - | uptime | + | copy_if_exists /user/basic_station/etc/ ${tmpdir}/user/basic_station/ |
| - | echo -e "\n\n\n************** cat /etc/version *****************" | + | copy_if_exists /user/basic_station/log/ ${tmpdir}/user/basic_station/ |
| - | cat /etc/version | + | fi |
| - | echo -e "\n\n\n************** ps w *****************" | + | copy_if_exists /etc/station/ ${tmpdir}/etc/ |
| - | ps w | + | |
| - | echo -e "\n\n\n************** opkg list-installed *****************" | + | |
| - | opkg list-installed | + | |
| - | echo -e "\n\n\n************** opkg list *****************" | + | |
| - | opkg list | + | |
| - | echo -e "\n\n\n************** monit status *****************" | + | |
| - | monit status | + | |
| - | echo -e "\n\n\n************** mount *****************" | + | |
| - | mount | + | |
| - | ) >> ${tmpdir}/cmd_results.txt | + | |
| - | ( | + | # miscellaneous configuration and logs |
| - | echo -e "\n\n\n************** ifconfig -a *****************" | + | echo "Extracting miscellaneous configuration and logs" |
| - | ifconfig -a | + | copy_if_exists /etc/default/ ${tmpdir}/etc/ |
| - | echo -e "\n\n\n************** gsmdiag.py *****************" | + | copy_if_exists /etc/openvpn/ ${tmpdir}/etc/ |
| - | gsmdiag.py ; cat /tmp/gsmdiag.txt | + | copy_if_exists /.update/update.log ${tmpdir}/ |
| - | echo -e "\n\n\n************** route *****************" | + | copy_if_exists /user/initrd/initrd.log ${tmpdir}/user/initrd |
| - | route | + | copy_if_exists /user/.update ${tmpdir}/user/ |
| - | echo -e "\n\n\n************** iptables -L *****************" | + | copy_if_exists /user/.upgrade.log ${tmpdir}/user/ |
| - | iptables -L | + | copy_if_exists /user/.sysupgrade.log ${tmpdir}/user/ |
| - | echo -e "\n\n\n************** connmanctl technologies *****************" | + | copy_if_exists /etc/version ${tmpdir}/etc/ |
| - | connmanctl technologies | + | copy_if_exists /etc/sysupgrade.conf ${tmpdir}/etc/ |
| - | echo -e "\n\n\n************** services *****************" | + | copy_if_exists /etc/sysupgrade.d/ ${tmpdir}/etc/ |
| - | connmanctl services | + | copy_if_exists /etc/.squash_layer_version ${tmpdir}/etc/ |
| + | echo ${SCRIPT_VERSION} > ${tmpdir}/script_version | ||
| - | if [ ${platform} == "IBTS" ]; then | + | if ${EXEC_COMMANDS} ; then |
| - | echo -e "\n\n\n************** qmicli get mode *****************" | + | execute_commands |
| - | qmicli -d /dev/cdc-wdm1 --nas-get-system-selection-preference | + | fi |
| - | fi | + | |
| - | ) >> ${tmpdir}/network_result.txt | + | |
| - | + | ||
| - | # get the 25 last NMEA frames; it will take 10s + 25s | + | |
| - | if [ ${platform} == "IBTS" ]; then | + | |
| - | + | ||
| - | if [ -n "$(lsof | grep $(readlink /dev/nmea2))" ]; then | + | |
| - | echo " /dev/nmea2 already used by $(lsof | grep $(readlink /dev/nmea2))" > ${tmpdir}/nmea2.log; | + | |
| - | + | ||
| - | else | + | |
| - | timeout -t 10 cat /dev/nmea2 > /dev/null | + | |
| - | timeout -t 40 head -n 50 /dev/nmea2 > ${tmpdir}/nmea2.log | + | |
| - | fi | + | |
| - | fi | + | |
| - | # create an archive with all the files and command results | + | # Finish by system logs to prevent space errors |
| - | # the archive is created on the USB key | + | echo "Extracting system logs" |
| - | tar -C /tmp -czf ${1}/${NAME}.tar.gz ${NAME} | + | if [ "${option}" = "minimal" ]; then |
| - | echo "${1}/${NAME}.tar.gz" | + | copy_if_exists /var/log/messages ${tmpdir}/var/log/ |
| - | sync | + | copy_if_exists /var/log/dmesg ${tmpdir}/var/log/ |
| + | for filename in /var/log/*.1; do | ||
| + | copy_if_exists ${filename} ${tmpdir}/var/log/ | ||
| + | done | ||
| + | for filename in /var/log/*.log; do | ||
| + | copy_if_exists ${filename} ${tmpdir}/var/log/ | ||
| + | done | ||
| + | else | ||
| + | copy_if_exists /var/log/ ${tmpdir}/var/ | ||
| + | fi | ||
| - | rm -rf $tmpdir | + | echo "Saving all data in ${output_dir}/${name}.tar.gz" |
| + | tar -C /user -czf ${output_dir}/${name}.tar.gz ${name} | ||
| + | echo "Done" | ||
| + | sync | ||
| + | |||
| + | rm -rf $tmpdir | ||
| } | } | ||
| Line 430: | Line 1060: | ||
| ##################### | ##################### | ||
| - | check_version | + | # Detect the environment of execution of the script |
| + | detect_env | ||
| - | platform=$(get_platform) | + | # Manage scripts arguments |
| + | while [[ $# -gt 0 ]] | ||
| + | do | ||
| + | key="$1" | ||
| - | reg=$(fslotpreg read GP1) | + | case $key in |
| - | if [ $? -eq 0 ]; then | + | -l|--ignore-leds) |
| - | serial=$(printf "%X\n" $reg) | + | LEDS_MANAGEMENT=false |
| - | else | + | shift |
| - | serial="unknown" | + | ;; |
| - | fi | + | -o|--output-dir) |
| - | shortmac=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}') | + | OUTPUT_DIR="$2" |
| + | shift 2 | ||
| + | ;; | ||
| + | -c|--disable-commands) | ||
| + | EXEC_COMMANDS=false | ||
| + | shift | ||
| + | ;; | ||
| + | -h|--help) | ||
| + | show_help | ||
| + | shift | ||
| + | exit 1 | ||
| + | ;; | ||
| + | -v|--version) | ||
| + | show_version | ||
| + | shift | ||
| + | exit 1 | ||
| + | ;; | ||
| + | -m|--minimal) | ||
| + | OPTION="minimal" | ||
| + | shift | ||
| + | ;; | ||
| + | *) | ||
| + | shift | ||
| + | ;; | ||
| + | esac | ||
| + | done | ||
| - | if [ ${platform} == "IBTS" ]; then | + | # Check if firmware version is compatible |
| - | SYSLEDPATH=${SYSLEDPATH_IBTS} | + | check_fw_version |
| - | elif [ ${platform} == "IFEMTO" ]; then | + | |
| - | SYSLEDPATH=${SYSLEDPATH_IFEMTO} | + | # Detect type of platform |
| - | fi | + | detect_platform |
| + | |||
| + | # Fill SERIAL and SHORT_MAC variables | ||
| + | fill_serials | ||
| # Set led On | # Set led On | ||
| - | set_led none 1 | + | if ${LEDS_MANAGEMENT} ; then |
| + | detect_leds | ||
| + | set_led none 1 | ||
| + | fi | ||
| # Get logs | # Get logs | ||
| - | get_logs ${USBDIR} | + | get_logs ${OUTPUT_DIR} ${OPTION} |
| - | + | ||
| - | # Prepare reboot after USB unplug | + | |
| - | cat > /tmp/usbkey_remove << EOF | + | |
| + | # End of script, manage user's notifications | ||
| + | if ${LEDS_MANAGEMENT} ; then | ||
| + | # Prepare reboot after USB unplug | ||
| + | cat > /tmp/usbkey_remove << EOF | ||
| # Switch off Led | # Switch off Led | ||
| - | echo "none" > ${SYSLEDPATH}/trigger | + | echo "none" > ${LED_PATH}/trigger |
| EOF | EOF | ||
| - | # All is right, make Led blinking fast to notify User | + | # All is right, make Led blinking fast to notify User |
| - | set_led timer 100 100 | + | set_led timer 100 100 |
| + | fi | ||
| </code> | </code> | ||
| ++++ | ++++ | ||
| + | ===== Helium ===== | ||
| + | This section contains some tips to do the first level of analysis (support level 1) on Helium gateways.\\ | ||
| + | It will help you to investigate before escalate to Kerlink support (level 2). | ||
| + | |||
| + | |||
| + | **Please note that there is an existing open wiki for Helium users: [[https://wikikerlink.fr/helium|Kerlink Helium Hotspot Wiki]]**\\ | ||
| + | * First step is to follow [[https://wikikerlink.fr/helium/doku.php?id=home#troubleshooting_guide|Troubleshooting guide]] chapter which contains: | ||
| + | * [[https://wikikerlink.fr/helium/doku.php?id=home#self_diagnostic|Self diagnostic]] section | ||
| + | * [[https://wikikerlink.fr/helium/doku.php?id=home#useful_links|Useful links]] section | ||
| + | |||
| + | * Second step, if necessary, is to follow the table below: | ||
| + | |||
| + | ^ Source of info ^ KPI ^ Value ^ Action ^ | ||
| + | ^ [[https://helium-onboarding.kerlink.com/|Onboarding dashboard]] | Last seen | More than 30m ago | Gateway is offline. Ask customer to check power/internet. | | ||
| + | | ::: | ::: | Less than 30m ago | Gateway is online. Check next KPI. | | ||
| + | | ::: | Block height | = 0 | Gateway is downloading a snapshot. | | ||
| + | | ::: | ::: | = 1 | Gateway is loading the downloaded snapshot. | | ||
| + | | ::: | ::: | Else | KPI is ok, check next KPI. | | ||
| + | | ::: | Block age | > 30 mn | Gateway is syncing, check again in one hour. If it persists, you can open a ticket to Kerlink Support. | | ||
| + | | ::: | ::: | < 30 mn | Gateway is synced. Check next KPI. | | ||
| + | ^ [[https://explorer.helium.com//|Helium explorer]] | 24h/14d earnings | > 0 HNT | Seems OK. Confirm with other Explorer KPIs. | | ||
| + | | ::: | Last activity date | < 3h | Seems OK. Confirm with other Explorer KPIs. | | ||
| + | | ::: | Is there a constructed challenge in the past 24h? | Yes | Seems OK. Confirm with other Explorer KPIs. | | ||
| + | | ::: | ::: | No | When gateway is synced, it should challenge. You can open a ticket to Kerlink Support. | | ||
| + | | ::: | Is there a challenged beaconer in the past 24h? | Yes | Seems OK. Confirm with other Explorer KPIs. | | ||
| + | | ::: | ::: | No | Gateway might be relayed. If not, you can open a ticket to Kerlink Support. | | ||
| + | | ::: | Is there a witnessed beacon in the past 8-24h? | Value depends on density around miner | Seems OK. Confirm with other Explorer KPIs. | | ||
| + | | ::: | ::: | No | You can open a ticket to Kerlink Support. | | ||
| + | | ::: | Is there a “broadcasted beacon” in the past 5d? | Yes | Seems OK. Confirm with other Explorer KPIs. | | ||
| + | | ::: | ::: | No | This is random but it seems odd. You can open a ticket to Kerlink Support. | | ||
| + | |||
| + | ==== Gateways connected to a WMC==== | ||
| + | For gateways connected to a WMC, you can use the Remote Command: Remote Access > Command\\ | ||
| + | {{:images:remote_command_wanesy_management_center.png?800|}} | ||
| + | <note important>** Due its cryptographic key and to ensure Kerlink Helium Hotspot security, no SSH access can be given to those gateways. \\ **</note> | ||
| + | |||
| + | You can use commands below: | ||
| + | ^ Command ^ Awaited result ^ | ||
| + | | opkg list | {{:images:opkg.png?400|}} | | ||
| + | | monit status miner | {{:images:monit_status_miner.png?400|}} |Status = OK | | ||
| + | | monit status lorafwd_helium | {{:images:monit_status_lorafwd_helium.png?400|}} | Status = OK | | ||
| + | | /etc/init.d/miner status | {{:images:miner_status.png?400|}} | Check Diff | | ||
| + | If the result is different than the awaited result, you can open a ticket to Kerlink Support. | ||
wiki/support/troubleshoot.1569419836.txt.gz · Last modified: 2019/09/25 15:57 by ghi
