wiki:support:troubleshoot
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
wiki:support:troubleshoot [2020/03/18 17:16] bdu [Automatic log gathering] |
wiki:support:troubleshoot [2023/06/02 15:47] (current) ehe [Automatic log gathering] |
||
|---|---|---|---|
| Line 11: | Line 11: | ||
| * ''/tmp/board_info.json'': This file contains information about the hardware of the gateway. It is generated at each boot time.\\ | * ''/tmp/board_info.json'': This file contains information about the hardware of the gateway. It is generated at each boot time.\\ | ||
| * ''/tmp/calib_loraloc.json'' (**for Wirnet iBTS**): This file contains the calibration values specific to this gateway. It is generated at boot time. | * ''/tmp/calib_loraloc.json'' (**for Wirnet iBTS**): This file contains the calibration values specific to this gateway. It is generated at boot time. | ||
| - | * ''/tmp/calib_rf.json'' (**for Wirnet iFemtoCell, Wirnet iFemtoCell-Evolution and Wirnet iStation**): This file contains the calibration values specific to this gateway. It is generated at boot time. | + | * ''/tmp/calib_rf.json'' (**for Wirnet iFemtoCell, Wirnet iFemtoCell-evolution and Wirnet iStation**): This file contains the calibration values specific to this gateway. It is generated at boot time. |
| * ''/dev/nmea*'' (**Wirnet iBTS and Wirnet iStation**): These devices receive NMEA frames (GPS) every seconds. It can be monitored with the ''cat'' command. | * ''/dev/nmea*'' (**Wirnet iBTS and Wirnet iStation**): These devices receive NMEA frames (GPS) every seconds. It can be monitored with the ''cat'' command. | ||
| Line 66: | Line 66: | ||
| * For Wirnet iFemtoCell, the serial number is written on the rear sticker (it is named "Final product ID"). | * For Wirnet iFemtoCell, the serial number is written on the rear sticker (it is named "Final product ID"). | ||
| * For Wirnet iStation, the serial number is written on the external sticker (it is named "Product ID"). | * For Wirnet iStation, the serial number is written on the external sticker (it is named "Product ID"). | ||
| - | * For Wirnet iFemtoCell-Evolution, the serial number is written on the external sticker (it is named "Product ID"). | + | * For Wirnet iFemtoCell-evolution, the serial number is written on rear sticker (it is named "Product ID"). |
| * A precise description of your problem. "Gateway not working" is not enough to investigate. | * A precise description of your problem. "Gateway not working" is not enough to investigate. | ||
| * The list of the actions you took to troubleshoot/investigate your problem. | * The list of the actions you took to troubleshoot/investigate your problem. | ||
| Line 78: | Line 78: | ||
| To simplify the log gathering process, two scripts are available. These scripts generate an archive containing many logs and commands results. | To simplify the log gathering process, two scripts are available. These scripts generate an archive containing many logs and commands results. | ||
| - | ++++Gathering logs from shell| | + | ++++Gathering logs from WebUI since release 5.9| |
| + | [[wiki:systeme_mana:webui#gateway_administration_menu|Get logs from gateway]] | ||
| + | ++++ | ||
| + | |||
| + | ++++Gathering logs from shell since release 4.2| | ||
| To gather the logs from shell: | To gather the logs from shell: | ||
| - | * Upload the script below (use scp, FileZilla, WinSCP, copy/paste, ...) in a folder of the gateway and go to this folder (for example in ''/user'' and issue ''cd /user''). | + | * Execute the command ''get_logs''. //The execution of the script takes less than 1 minute.// |
| - | * Convert the file to Unix text file format using ''dos2unix gather_logs.sh''. | + | * It will generate an archive in the same directory you executed the script (example: ''Logs_2E0605F5_7b26_20200124-095206.tar.gz'') |
| - | * Add execution rights to the script using ''chmod 755 gather_logs.sh''. | + | * Retrieve this archive (use whatever method you prefer: FileZilla, scp, ...). |
| - | * Execute the script using ''./gather_logs.sh''. //The execution of the script takes about 1-2 minutes.// | + | |
| - | * It will generate an archive under the same directory you executed the script. | + | <note>**To use only in case of speed or data restriction**, there is a new option to gather lighter logs: ''get_logs -m'' or ''get_logs --minimal''. \\ |
| - | * Retrieve this archive (use whatever method you prefer: scp, FileZilla, WinSCP, ...). | + | **Please note that it will only gather latest logs** (.log & .log.1 files). \\ |
| + | It will generate an light archive recognizable by its name: _minimal.tar.gz (example: ''Logs_3903002E_0091_20230419-134556_minimal.tar.gz'') | ||
| + | </note> | ||
| + | ++++ | ||
| + | |||
| + | ++++Gathering logs from shell on older releases| | ||
| + | |||
| + | To gather the logs from shell: | ||
| + | * Upload the script bellow (use whatever method you prefer: FileZilla, scp, copy/paste, ...) in a folder of your gateway and go to this folder. | ||
| + | * Convert the file to Unix text file format using ''dos2unix get_logs.sh''. | ||
| + | * Add execution rights to the script using ''chmod 755 get_logs.sh''. | ||
| + | * Execute the script using ''./get_logs.sh''. //The execution of the script takes less than 1 minute.// | ||
| + | * It will generate an archive in the same directory you executed the script. (example: ''Logs_2E0605F5_7b26_20200124-095206.tar.gz'') | ||
| + | * Retrieve this archive (use whatever method you prefer: FileZilla, scp, ...). | ||
| - | <code bash gather_logs.sh> | + | <code bash get_logs.sh> |
| #!/bin/sh | #!/bin/sh | ||
| Line 317: | Line 335: | ||
| } | } | ||
| + | get_nmea_frames() { | ||
| + | gps_tty="$1" | ||
| + | |||
| + | if ! [ -e "$gps_tty" ]; then | ||
| + | echo "$gps_tty doesn't exist" | ||
| + | return 1 | ||
| + | fi | ||
| + | |||
| + | if [ -n "$(lsof | grep $(readlink $gps_tty))" ]; then | ||
| + | echo "$gps_tty already used by $(lsof | grep $(readlink $gps_tty))" | ||
| + | return 2 | ||
| + | else | ||
| + | # Pseudo-term keep only FIRST frames so we keep last of these old frames and get few new frames | ||
| + | # It will take 3 seconds | ||
| + | timeout -t 3 cat $gps_tty > /tmp/nmea.log | ||
| + | cat /tmp/nmea.log | ||
| + | fi | ||
| + | } | ||
| # | # | ||
| Line 351: | Line 387: | ||
| gsmdiag.py ; cat /tmp/gsmdiag.txt | gsmdiag.py ; cat /tmp/gsmdiag.txt | ||
| echo -e "\n\n\n************** route *****************" | echo -e "\n\n\n************** route *****************" | ||
| - | route | + | route -n |
| echo -e "\n\n\n************** iptables -L *****************" | echo -e "\n\n\n************** iptables -L *****************" | ||
| iptables -L | iptables -L | ||
| Line 365: | Line 401: | ||
| ) >> ${tmpdir}/network_result.txt | ) >> ${tmpdir}/network_result.txt | ||
| - | # get the 25 last NMEA frames; it will take 10s + 25s | + | get_nmea_frames /dev/nmea3 > ${tmpdir}/nmea.log |
| - | if [ -e /dev/nmea2 ]; then | + | [ $? -ne 0 ] && get_nmea_frames /dev/nmea2 >> ${tmpdir}/nmea.log |
| - | echo -e "\tGPS commands" | + | |
| - | if [ -n "$(lsof | grep $(readlink /dev/nmea2))" ]; then | + | |
| - | echo " /dev/nmea2 already used by $(lsof | grep $(readlink /dev/nmea2))" > ${tmpdir}/nmea2.log; | + | |
| - | else | + | |
| - | timeout -t 10 cat /dev/nmea2 > /dev/null | + | |
| - | timeout -t 40 head -n 50 /dev/nmea2 > ${tmpdir}/nmea2.log | + | |
| - | fi | + | |
| - | fi | + | |
| # Overlay modifications | # Overlay modifications | ||
| Line 396: | Line 424: | ||
| local output_dir="$1" | local output_dir="$1" | ||
| local name="Logs_${SERIAL}_${SHORT_MAC}_$(date "+%Y%m%d-%H%M%S")" | local name="Logs_${SERIAL}_${SHORT_MAC}_$(date "+%Y%m%d-%H%M%S")" | ||
| - | local tmpdir="/tmp/${name}" | + | local tmpdir="/user/${name}" |
| - | + | ||
| - | # system logs | + | |
| - | echo "Extracting system logs" | + | |
| - | copy_if_exists /var/log/ ${tmpdir}/var/ | + | |
| # tmp files | # tmp files | ||
| Line 449: | Line 473: | ||
| echo "Extracting miscellaneous configuration and logs" | echo "Extracting miscellaneous configuration and logs" | ||
| copy_if_exists /etc/rcU.d/ ${tmpdir}/etc/ | copy_if_exists /etc/rcU.d/ ${tmpdir}/etc/ | ||
| + | copy_if_exists /.update/update.log ${tmpdir}/ | ||
| copy_if_exists /user/.update ${tmpdir}/user/ | copy_if_exists /user/.update ${tmpdir}/user/ | ||
| copy_if_exists /user/.upgrade.log ${tmpdir}/user/ | copy_if_exists /user/.upgrade.log ${tmpdir}/user/ | ||
| Line 461: | Line 486: | ||
| execute_commands | execute_commands | ||
| fi | fi | ||
| + | |||
| + | # Finish by system logs to prevent space errors | ||
| + | echo "Extracting system logs" | ||
| + | copy_if_exists /var/log/ ${tmpdir}/var/ | ||
| echo "Saving all data in ${output_dir}/${name}.tar.gz" | echo "Saving all data in ${output_dir}/${name}.tar.gz" | ||
| - | tar -C /tmp -czf ${output_dir}/${name}.tar.gz ${name} | + | tar -C /user -czf ${output_dir}/${name}.tar.gz ${name} |
| echo "Done" | echo "Done" | ||
| sync | sync | ||
| Line 542: | Line 571: | ||
| fi | fi | ||
| </code> | </code> | ||
| + | |||
| + | |||
| ++++ | ++++ | ||
| Line 548: | Line 579: | ||
| To gather logs from a USB without typing any command: | To gather logs from a USB without typing any command: | ||
| - | * Download the following script and copy it on a USB key. | + | * Download the following script and copy it on a USB flash drive. |
| - | * Create a ''usbkey.txt'' file on the USB key as described [[wiki:keros_custo:sw_updates#specific_files|here]]. | + | * Create a ''usbkey.txt'' file on the USB flash drive as described [[wiki:keros_custo:sw_updates#specific_files|here]]. |
| - | * Plug the USB key on the CPU board. | + | * Plug the USB flash drive into the USB port of the gateway. |
| * Wait for the yellow debug led fast blinking. //The execution of the script takes about 1-2 minutes.// | * Wait for the yellow debug led fast blinking. //The execution of the script takes about 1-2 minutes.// | ||
| - | * Unplug the USB key. | + | * Unplug the USB flash drive. |
| - | * An archive with all the required logs has been created on the USB key. | + | * An archive with all the required logs has been created on the USB flash drive. |
| <code bash usb.autorun> | <code bash usb.autorun> | ||
| Line 559: | Line 591: | ||
| # Version of the script | # Version of the script | ||
| - | SCRIPT_VERSION=1.3 | + | SCRIPT_VERSION=1.4 |
| # Output directory for the result of the script | # Output directory for the result of the script | ||
| Line 588: | Line 620: | ||
| # | # | ||
| show_version() { | show_version() { | ||
| - | echo "Script version: ${SCRIPT_VERSION}" | + | echo "Script version: ${SCRIPT_VERSION}" |
| } | } | ||
| Line 597: | Line 629: | ||
| show_help() | show_help() | ||
| { | { | ||
| - | show_version | + | show_version |
| - | echo -e "Usage: $0 [-h|--help] [-l|--ignore-leds] [-o|--output-dir <directory>] | + | echo -e "Usage: $0 [-h|--help] [-l|--ignore-leds] [-o|--output-dir <directory>] |
| \t-h|--help: this help | \t-h|--help: this help | ||
| \t-v|--version: version of the script | \t-v|--version: version of the script | ||
| Line 604: | Line 636: | ||
| \t-o|--output-dir <directory>: force the output <directory> to use for extraction (default: current directory) | \t-o|--output-dir <directory>: force the output <directory> to use for extraction (default: current directory) | ||
| \t-c|--disable-commands: disable diagnosis commands (default: executed) | \t-c|--disable-commands: disable diagnosis commands (default: executed) | ||
| + | \t-m|--minimal: get minimal logs (to lower generated file size) | ||
| " | " | ||
| } | } | ||
| Line 614: | Line 647: | ||
| # | # | ||
| compare_versions () { | compare_versions () { | ||
| - | if [[ $1 == $2 ]] | + | if [[ $1 == $2 ]] |
| - | then | + | then |
| - | return 0 | + | return 0 |
| - | fi | + | fi |
| - | local IFS=. | + | local IFS=. |
| - | local i ver1=($1) ver2=($2) | + | local i ver1=($1) ver2=($2) |
| - | # fill empty fields in ver1 with zeros | + | # fill empty fields in ver1 with zeros |
| - | for ((i=${#ver1[@]}; i<${#ver2[@]}; i++)) | + | for ((i=${#ver1[@]}; i<${#ver2[@]}; i++)); do |
| - | do | + | ver1[i]=0 |
| - | ver1[i]=0 | + | done |
| - | done | + | for ((i=0; i<${#ver1[@]}; i++)); do |
| - | for ((i=0; i<${#ver1[@]}; i++)) | + | if [[ -z ${ver2[i]} ]] |
| - | do | + | then |
| - | if [[ -z ${ver2[i]} ]] | + | # fill empty fields in ver2 with zeros |
| - | then | + | ver2[i]=0 |
| - | # fill empty fields in ver2 with zeros | + | fi |
| - | ver2[i]=0 | + | if ((10#${ver1[i]} > 10#${ver2[i]})) |
| - | fi | + | then |
| - | if ((10#${ver1[i]} > 10#${ver2[i]})) | + | return 1 |
| - | then | + | fi |
| - | return 1 | + | if ((10#${ver1[i]} < 10#${ver2[i]})) |
| - | fi | + | then |
| - | if ((10#${ver1[i]} < 10#${ver2[i]})) | + | return 2 |
| - | then | + | fi |
| - | return 2 | + | done |
| - | fi | + | return 0 |
| - | done | + | |
| - | return 0 | + | |
| } | } | ||
| Line 652: | Line 683: | ||
| # | # | ||
| copy_if_exists(){ | copy_if_exists(){ | ||
| - | local from="$1" | + | local from="$1" |
| - | local to="$2" | + | local to="$2" |
| - | if [ -e "${from}" ] ; then | + | if [ -e "${from}" ] ; then |
| - | [ ! -d "${to}" ] && mkdir -p "${to}" | + | [ ! -d "${to}" ] && mkdir -p "${to}" |
| - | cp -a "${from}" "${to}" | + | cp -a "${from}" "${to}" |
| - | fi | + | fi |
| } | } | ||
| Line 668: | Line 699: | ||
| # | # | ||
| check_fw_version(){ | check_fw_version(){ | ||
| - | # check if /etc/version exists | + | # check if /etc/version exists |
| - | if [ -a /etc/version ]; then | + | if [ -a /etc/version ]; then |
| - | FW_VERSION=$(cat /etc/version) | + | FW_VERSION=$(cat /etc/version) |
| - | # Detect type of firmware: KerOS or SPN | + | # Detect type of firmware: KerOS or SPN |
| - | if [[ "${FW_VERSION}" == *-spn-* ]]; then | + | if [[ "${FW_VERSION}" == *-spn-* ]]; then |
| - | # Check if SPN version is not inferior to 2.1, stop if it is the case | + | # Check if SPN version is not inferior to 2.1, stop if it is the case |
| - | compare_versions ${FW_VERSION:0:3} "2.1" | + | compare_versions ${FW_VERSION:0:3} "2.1" |
| - | if [ $? -eq 2 ]; then | + | if [ $? -eq 2 ]; then |
| - | echo "Your SPN firmware version ($FW_VERSION) is inferior to v2.1. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt | + | echo "Your SPN firmware version ($FW_VERSION) is inferior to v2.1. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt |
| - | exit | + | exit |
| - | fi | + | fi |
| - | else | + | else |
| - | # Check if KerOS version is not inferior to 4.0, stop if it is the case | + | # Check if KerOS version is not inferior to 4.0, stop if it is the case |
| - | compare_versions ${FW_VERSION:0:3} "4.0" | + | compare_versions ${FW_VERSION:0:3} "4.0" |
| - | if [ $? -eq 2 ]; then | + | if [ $? -eq 2 ]; then |
| - | echo "Your KerOS firmware version ($FW_VERSION) is inferior to v4. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt | + | echo "Your KerOS firmware version ($FW_VERSION) is inferior to v4. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt |
| - | exit | + | exit |
| - | fi | + | fi |
| - | fi | + | fi |
| - | else | + | else |
| - | echo "Could not find /etc/version" >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt | + | echo "Could not find /etc/version" >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt |
| - | exit | + | exit |
| - | fi | + | fi |
| } | } | ||
| Line 699: | Line 730: | ||
| # | # | ||
| detect_platform() { | detect_platform() { | ||
| - | PLATFORM="$(fslotpreg getplatform)" | + | PLATFORM="$(fslotpreg getplatform)" |
| - | if [ $? -ne 0 ]; | + | if [ $? -ne 0 ]; |
| - | then | + | then |
| - | PLATFORM="NOT_SUPPORTED" | + | PLATFORM="NOT_SUPPORTED" |
| - | fi | + | fi |
| - | echo "Platform detected: ${PLATFORM}" | + | echo "Platform detected: ${PLATFORM}" |
| } | } | ||
| Line 712: | Line 743: | ||
| # | # | ||
| fill_serials() { | fill_serials() { | ||
| - | local reg=$(fslotpreg read GP1) | + | local reg=$(fslotpreg read GP1) |
| - | if [ $? -eq 0 ]; then | + | if [ $? -eq 0 ]; then |
| - | SERIAL=$(printf "%X\n" $reg) | + | SERIAL=$(printf "%X\n" $reg) |
| - | else | + | else |
| - | SERIAL="unknown" | + | SERIAL="unknown" |
| - | fi | + | fi |
| - | SHORT_MAC=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}') | + | if [ -f "/sys/class/net/eth0/address" ]; then |
| + | SHORT_MAC=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}') | ||
| + | elif [ -f "/sys/class/net/usb0/address" ]; then | ||
| + | SHORT_MAC=$(cat /sys/class/net/usb0/address | awk -F: '{print $5$6}') | ||
| + | else | ||
| + | SHORT_MAC="XXXX" | ||
| + | fi | ||
| } | } | ||
| Line 729: | Line 766: | ||
| # | # | ||
| detect_env() { | detect_env() { | ||
| - | # Detection is only based on the presence or not of the variable DEVNAME | + | # Detection is only based on the presence or not of the variable DEVNAME |
| - | if [ ! -z ${DEVNAME} ] ; then | + | if [ ! -z ${DEVNAME} ] ; then |
| - | SCRIPT_ENV=udev | + | SCRIPT_ENV=udev |
| - | exec > /dev/console 2>&1 # Output the script execution in debug console | + | exec > /dev/console 2>&1 # Output the script execution in debug console |
| - | else | + | else |
| - | SCRIPT_ENV=shell | + | SCRIPT_ENV=shell |
| - | LEDS_MANAGEMENT=false | + | LEDS_MANAGEMENT=false |
| - | OUTPUT_DIR=$(pwd) | + | OUTPUT_DIR=$(pwd) |
| - | fi | + | fi |
| - | echo "Script environment: ${SCRIPT_ENV}" | + | echo "Script environment: ${SCRIPT_ENV}" |
| } | } | ||
| Line 746: | Line 783: | ||
| # | # | ||
| detect_leds() { | detect_leds() { | ||
| - | if [ -e "${SYSLEDPATH}" ]; then | + | if [ -e "${SYSLEDPATH}" ]; then |
| - | LED_PATH="${SYSLEDPATH}" | + | LED_PATH="${SYSLEDPATH}" |
| - | elif [ -e "${SYSLEDPATH_4_0_IBTS}" ]; then | + | elif [ -e "${SYSLEDPATH_4_0_IBTS}" ]; then |
| - | LED_PATH="${SYSLEDPATH_4_0_IBTS}" | + | LED_PATH="${SYSLEDPATH_4_0_IBTS}" |
| - | elif [ -e "${SYSLEDPATH_4_0_IFEMTO}" ]; then | + | elif [ -e "${SYSLEDPATH_4_0_IFEMTO}" ]; then |
| - | LED_PATH="${SYSLEDPATH_4_0_IFEMTO}" | + | LED_PATH="${SYSLEDPATH_4_0_IFEMTO}" |
| - | fi | + | fi |
| } | } | ||
| Line 763: | Line 800: | ||
| # | # | ||
| set_led() { | set_led() { | ||
| - | trigger="${1}" | + | trigger="${1}" |
| - | on="$2" | + | on="$2" |
| - | off="$3" | + | off="$3" |
| - | if [ -e "${LED_PATH}" ]; then | + | if [ -e "${LED_PATH}" ]; then |
| - | echo $trigger > "${LED_PATH}/trigger" | + | echo $trigger > "${LED_PATH}/trigger" |
| - | case "$trigger" in | + | case "$trigger" in |
| - | "timer") | + | "timer") |
| - | echo ${on:=500} > "${LED_PATH}/delay_on" | + | echo ${on:=500} > "${LED_PATH}/delay_on" |
| - | echo ${off:=500} > "${LED_PATH}/delay_off" | + | echo ${off:=500} > "${LED_PATH}/delay_off" |
| - | ;; | + | ;; |
| - | "none") | + | "none") |
| - | echo ${on:=0} > "${LED_PATH}/brightness" | + | echo ${on:=0} > "${LED_PATH}/brightness" |
| - | ;; | + | ;; |
| - | *) | + | *) |
| - | # Nothing to do | + | # Nothing to do |
| - | ;; | + | ;; |
| - | esac | + | esac |
| - | fi | + | fi |
| } | } | ||
| + | get_nmea_frames() { | ||
| + | gps_tty="$1" | ||
| + | |||
| + | if ! [ -e "$gps_tty" ]; then | ||
| + | echo "$gps_tty doesn't exist" | ||
| + | return 1 | ||
| + | fi | ||
| + | |||
| + | if [ -n "$(lsof | grep $(readlink $gps_tty))" ]; then | ||
| + | echo "$gps_tty already used by $(lsof | grep $(readlink $gps_tty))" | ||
| + | return 2 | ||
| + | else | ||
| + | # Pseudo-term keep only FIRST frames so we keep last of these old frames and get few new frames | ||
| + | # It will take 3 seconds | ||
| + | timeout -t 3 cat $gps_tty > /tmp/nmea.log | ||
| + | cat /tmp/nmea.log | ||
| + | fi | ||
| + | } | ||
| # | # | ||
| Line 789: | Line 844: | ||
| # | # | ||
| execute_commands() { | execute_commands() { | ||
| - | echo "Executing diagnosis commands (can take a long time)" | + | echo "Executing diagnosis commands (can take a long time)" |
| - | echo -e "\tSystems commands" | + | echo -e "\tSystems commands" |
| - | ( | + | ( |
| - | echo -e "\n\n\n************** fslotpreg getplatform *****************" | + | echo -e "\n\n\n************** fslotpreg getplatform *****************" |
| - | fslotpreg getplatform | + | fslotpreg getplatform |
| - | echo -e "\n\n\n************** fslotpreg fslotpreg read GP1 *****************" | + | echo -e "\n\n\n************** fslotpreg fslotpreg read GP1 *****************" |
| - | fslotpreg read GP1 | + | fslotpreg read GP1 |
| - | echo -e "\n\n\n************** date *****************" | + | echo -e "\n\n\n************** date *****************" |
| - | date | + | date |
| - | echo -e "\n\n\n************** uptime *****************" | + | echo -e "\n\n\n************** uptime *****************" |
| - | uptime | + | uptime |
| - | echo -e "\n\n\n************** ps ww *****************" | + | echo -e "\n\n\n************** ps ww *****************" |
| - | ps ww | + | ps ww |
| - | echo -e "\n\n\n************** opkg status *****************" | + | echo -e "\n\n\n************** opkg status *****************" |
| - | opkg status | + | opkg status |
| - | echo -e "\n\n\n************** monit status *****************" | + | echo -e "\n\n\n************** monit status *****************" |
| - | monit status | + | monit status |
| - | echo -e "\n\n\n************** mount *****************" | + | echo -e "\n\n\n************** mount *****************" |
| - | mount | + | mount |
| - | ) >> ${tmpdir}/cmd_results.txt | + | echo -e "\n\n\n************** OpenVPN Certificate expiration date *****************" |
| + | compare_versions ${FW_VERSION:0:3} "5.0" | ||
| + | ret=$? | ||
| + | if [ $ret -eq 1 ]; then | ||
| + | if [ -f "/etc/openvpn/bscc.p12" ]; then | ||
| + | openssl pkcs12 -info -in /etc/openvpn/bscc.p12 -nokeys -password file:/etc/openvpn/bscc.password| openssl x509 -noout -enddate | ||
| + | fi | ||
| + | elif [ $ret -eq 2 ]; then | ||
| + | grep -iA3 "Validity" /.update/update.log | tail -3 | grep -i "not after" | sed 's/^[ \t]*//' | ||
| + | fi | ||
| + | echo -e "\n\n\n************** Disk info (df -h) *****************" | ||
| + | df -h | ||
| + | echo -e "\n\n\n************** Log Size (du -sh /var/log) *****************" | ||
| + | du -sh /var/log | ||
| + | ) >> ${tmpdir}/cmd_results.txt | ||
| - | echo -e "\tNetwork commands" | + | echo -e "\tNetwork commands" |
| - | ( | + | ( |
| - | echo -e "\n\n\n************** ifconfig -a *****************" | + | echo -e "\n\n\n************** ifconfig -a *****************" |
| - | ifconfig -a | + | ifconfig -a |
| - | echo -e "\n\n\n************** gsmdiag.py *****************" | + | echo -e "\n\n\n************** gsmdiag.py *****************" |
| - | gsmdiag.py ; cat /tmp/gsmdiag.txt | + | gsmdiag.py ; cat /tmp/gsmdiag.txt |
| - | echo -e "\n\n\n************** route *****************" | + | echo -e "\n\n\n************** route *****************" |
| - | route | + | route -n |
| - | echo -e "\n\n\n************** iptables -L *****************" | + | echo -e "\n\n\n************** iptables -L *****************" |
| - | iptables -L | + | iptables -L |
| - | echo -e "\n\n\n************** connmanctl technologies *****************" | + | echo -e "\n\n\n************** connmanctl technologies *****************" |
| - | connmanctl technologies | + | connmanctl technologies |
| - | echo -e "\n\n\n************** services *****************" | + | echo -e "\n\n\n************** services *****************" |
| - | connmanctl services | + | connmanctl services |
| - | if [ -e /dev/cdc-wdm1 ]; then | + | if [ -e /dev/cdc-wdm1 ]; then |
| - | echo -e "\n\n\n************** qmicli get mode *****************" | + | echo -e "\n\n\n************** qmicli get mode *****************" |
| - | qmicli -d /dev/cdc-wdm1 --nas-get-system-selection-preference | + | qmicli -d /dev/cdc-wdm1 --nas-get-system-selection-preference |
| - | fi | + | fi |
| - | ) >> ${tmpdir}/network_result.txt | + | ) >> ${tmpdir}/network_result.txt |
| - | # get the 25 last NMEA frames; it will take 10s + 25s | + | get_nmea_frames /dev/nmea3 > ${tmpdir}/nmea.log |
| - | if [ -e /dev/nmea2 ]; then | + | [ $? -ne 0 ] && get_nmea_frames /dev/nmea2 >> ${tmpdir}/nmea.log |
| - | echo -e "\tGPS commands" | + | |
| - | if [ -n "$(lsof | grep $(readlink /dev/nmea2))" ]; then | + | |
| - | echo " /dev/nmea2 already used by $(lsof | grep $(readlink /dev/nmea2))" > ${tmpdir}/nmea2.log; | + | |
| - | else | + | |
| - | timeout -t 10 cat /dev/nmea2 > /dev/null | + | |
| - | timeout -t 40 head -n 50 /dev/nmea2 > ${tmpdir}/nmea2.log | + | |
| - | fi | + | |
| - | fi | + | |
| - | # Overlay modifications | + | # Overlay modifications |
| - | echo -e "\tOverlay commands" | + | echo -e "\tOverlay commands disabled (KEROS-1907)" |
| - | overlay diff -l /.rootfs.ro -u /user/.rootfs_upper/ >> ${tmpdir}/overlay_status.txt << EOF | + | # overlay diff -l /.rootfs.ro -u /user/.rootfs_upper/ >> ${tmpdir}/overlay_status.txt << EOF |
| - | yes | + | #yes |
| - | EOF | + | #EOF |
| - | echo "End of diagnosis commands" | + | echo "End of diagnosis commands" |
| } | } | ||
| Line 861: | Line 922: | ||
| get_logs() | get_logs() | ||
| { | { | ||
| - | local output_dir="$1" | + | local output_dir="$1" |
| - | local name="Logs_${SERIAL}_${SHORT_MAC}_$(date "+%Y%m%d-%H%M%S")" | + | local option="$2" |
| - | local tmpdir="/tmp/${name}" | + | local name="Logs_${SERIAL}_${SHORT_MAC}_$(date "+%Y%m%d-%H%M%S")" |
| + | if [ "${option}" = "minimal" ]; then | ||
| + | name="${name}_minimal" | ||
| + | fi | ||
| + | local tmpdir="/user/${name}" | ||
| - | # system logs | + | # tmp files |
| - | echo "Extracting system logs" | + | echo "Extracting tmp files" |
| - | copy_if_exists /var/log/ ${tmpdir}/var/ | + | copy_if_exists /tmp/sys_startup_status.json ${tmpdir}/tmp/ |
| + | copy_if_exists /tmp/board_info.json ${tmpdir}/tmp/ | ||
| + | copy_if_exists /tmp/calib_loraloc.json ${tmpdir}/tmp/ | ||
| + | copy_if_exists /tmp/calib_rf.json ${tmpdir}/tmp/ | ||
| - | # tmp files | + | # network configuration |
| - | echo "Extracting tmp files" | + | echo "Extracting network configuration" |
| - | copy_if_exists /tmp/sys_startup_status.json ${tmpdir}/tmp/ | + | if [ "${option}" = "minimal" ]; then |
| - | copy_if_exists /tmp/board_info.json ${tmpdir}/tmp/ | + | copy_if_exists /etc/network/connman/ ${tmpdir}/etc/network/ |
| - | copy_if_exists /tmp/calib_loraloc.json ${tmpdir}/tmp/ | + | copy_if_exists /etc/network/ofono/ ${tmpdir}/etc/network/ |
| - | copy_if_exists /tmp/calib_rf.json ${tmpdir}/tmp/ | + | copy_if_exists /etc/network/openvpn/ ${tmpdir}/etc/network/ |
| + | cp /etc/network/*.conf ${tmpdir}/etc/network/ | ||
| + | else | ||
| + | copy_if_exists /etc/network/ ${tmpdir}/etc/ | ||
| + | fi | ||
| + | copy_if_exists /etc/firewall.d/ ${tmpdir}/etc/ | ||
| - | # network configuration | + | # init configuration |
| - | echo "Extracting network configuration" | + | if [ "${option}" != "minimal" ]; then |
| - | copy_if_exists /etc/network/ ${tmpdir}/etc/ | + | echo "Extracting init.d and rc*.d files" |
| - | copy_if_exists /etc/firewall.d/ ${tmpdir}/etc/ | + | copy_if_exists /etc/init.d/ ${tmpdir}/etc/ |
| + | for filename in /etc/rc*.d; do | ||
| + | copy_if_exists ${filename} ${tmpdir}/etc/ | ||
| + | done | ||
| + | fi | ||
| - | # CPF (lorad + lorafwd) configuration | + | # CPF (lorad + lorafwd) configuration |
| - | echo "Extracting CPF configuration (if any)" | + | echo "Extracting CPF configuration (if any)" |
| - | for lorad_dir in /user/etc/lorad /etc/lorad | + | if [ "${option}" = "minimal" ]; then |
| - | do | + | copy_if_exists /user/etc/lorad/lorad.json ${tmpdir}/user/etc/lorad/ |
| - | if [ -d ${lorad_dir} ] ; then | + | copy_if_exists /etc/lorad/lorad.json ${tmpdir}/etc/lorad/ |
| - | echo -e "\tExtracting lorad configuration" | + | else |
| - | copy_if_exists /etc/default/lorad ${tmpdir}/etc/default/ | + | for lorad_dir in /user/etc/lorad /etc/lorad; do |
| - | copy_if_exists ${lorad_dir} ${tmpdir}${lorad_dir%/*} | + | if [ -d ${lorad_dir} ] ; then |
| - | fi | + | echo -e "\tExtracting lorad configuration" |
| - | done | + | copy_if_exists ${lorad_dir} ${tmpdir}${lorad_dir%/*} |
| - | for lorafwd_dir in /user/etc/lorafwd /etc/lorafwd | + | fi |
| - | do | + | done |
| - | if [ -d ${lorafwd_dir} ] ; then | + | fi |
| - | echo -e "\tExtracting lorafwd configuration" | + | copy_if_exists /etc/default/lorad ${tmpdir}/etc/default/ |
| - | copy_if_exists /etc/default/lorafwd ${tmpdir}/etc/default/ | + | |
| - | copy_if_exists ${lorafwd_dir} ${tmpdir}${lorafwd_dir%/*} | + | |
| - | fi | + | |
| - | done | + | |
| - | # WMC configuration and logs | + | echo -e "\tExtracting lorafwd configuration" |
| - | echo "Extracting WMC configuration and logs (if any)" | + | if [ "${option}" = "minimal" ]; then |
| - | if [ -d /user/bscc ] ; then | + | for filename in /etc/lorafwd/*.toml; do |
| - | echo -e "\tExtracting BSCC configuration and logs" | + | copy_if_exists ${filename} ${tmpdir}/etc/lorafwd/ |
| - | copy_if_exists /user/bscc/var.xml ${tmpdir}/user/bscc/ | + | done |
| - | copy_if_exists /user/bscc/traces/ ${tmpdir}/user/bscc/ | + | else |
| - | fi | + | for lorafwd_dir in /user/etc/lorafwd /etc/lorafwd; do |
| - | if [ -d /user/snmp ] ; then | + | if [ -d ${lorafwd_dir} ] ; then |
| - | echo -e "\tExtracting SNMP configuration and logs" | + | copy_if_exists ${lorafwd_dir} ${tmpdir}${lorafwd_dir%/*} |
| - | copy_if_exists /user/snmp/snmpd.conf ${tmpdir}/user/snmp/ | + | fi |
| - | copy_if_exists /user/snmp/traces/ ${tmpdir}/user/snmp/ | + | done |
| - | fi | + | fi |
| + | copy_if_exists /etc/lorafwd.toml ${tmpdir}/etc/ | ||
| + | copy_if_exists /etc/default/lorafwd ${tmpdir}/etc/default/ | ||
| - | # miscellaneous configuration and logs | + | # WMC configuration and logs |
| - | echo "Extracting miscellaneous configuration and logs" | + | echo "Extracting WMC configuration and logs (if any)" |
| - | copy_if_exists /etc/rcU.d/ ${tmpdir}/etc/ | + | if [ -d /user/bscc ] ; then |
| - | copy_if_exists /user/.update ${tmpdir}/user/ | + | echo -e "\tExtracting BSCC configuration and logs" |
| - | copy_if_exists /user/.upgrade.log ${tmpdir}/user/ | + | copy_if_exists /user/bscc/var.xml ${tmpdir}/user/bscc/ |
| - | copy_if_exists /user/.sysupgrade.log ${tmpdir}/user/ | + | copy_if_exists /user/bscc/traces/ ${tmpdir}/user/bscc/ |
| - | copy_if_exists /etc/version ${tmpdir}/etc/ | + | fi |
| - | copy_if_exists /etc/sysupgrade.conf ${tmpdir}/etc/ | + | if [ -d /user/snmp ] ; then |
| - | copy_if_exists /etc/sysupgrade.d/ ${tmpdir}/etc/ | + | echo -e "\tExtracting SNMP configuration and logs" |
| - | copy_if_exists /etc/.squash_layer_version ${tmpdir}/etc/ | + | copy_if_exists /user/snmp/snmpd.conf ${tmpdir}/user/snmp/ |
| - | echo ${SCRIPT_VERSION} > ${tmpdir}/script_version | + | copy_if_exists /user/snmp/traces/ ${tmpdir}/user/snmp/ |
| + | fi | ||
| + | copy_if_exists /etc/default/bscc ${tmpdir}/etc/default/ | ||
| + | copy_if_exists /etc/default/lora-snmp ${tmpdir}/etc/default/ | ||
| + | copy_if_exists /etc/snmp/snmpd.conf ${tmpdir}/etc/snmp/ | ||
| - | if ${EXEC_COMMANDS} ; then | + | # basic station configuration and logs |
| - | execute_commands | + | echo "Extracting basic station configuration and logs (if any)" |
| - | fi | + | if [ -d /user/basic_station ] ; then |
| + | echo -e "\tExtracting Basic Station configuration and logs" | ||
| + | copy_if_exists /user/basic_station/etc/ ${tmpdir}/user/basic_station/ | ||
| + | copy_if_exists /user/basic_station/log/ ${tmpdir}/user/basic_station/ | ||
| + | fi | ||
| + | copy_if_exists /etc/station/ ${tmpdir}/etc/ | ||
| - | echo "Saving all data in ${output_dir}/${name}.tar.gz" | + | # miscellaneous configuration and logs |
| - | tar -C /tmp -czf ${output_dir}/${name}.tar.gz ${name} | + | echo "Extracting miscellaneous configuration and logs" |
| - | echo "Done" | + | copy_if_exists /etc/default/ ${tmpdir}/etc/ |
| - | sync | + | copy_if_exists /etc/openvpn/ ${tmpdir}/etc/ |
| + | copy_if_exists /.update/update.log ${tmpdir}/ | ||
| + | copy_if_exists /user/initrd/initrd.log ${tmpdir}/user/initrd | ||
| + | copy_if_exists /user/.update ${tmpdir}/user/ | ||
| + | copy_if_exists /user/.upgrade.log ${tmpdir}/user/ | ||
| + | copy_if_exists /user/.sysupgrade.log ${tmpdir}/user/ | ||
| + | copy_if_exists /etc/version ${tmpdir}/etc/ | ||
| + | copy_if_exists /etc/sysupgrade.conf ${tmpdir}/etc/ | ||
| + | copy_if_exists /etc/sysupgrade.d/ ${tmpdir}/etc/ | ||
| + | copy_if_exists /etc/.squash_layer_version ${tmpdir}/etc/ | ||
| + | echo ${SCRIPT_VERSION} > ${tmpdir}/script_version | ||
| + | |||
| + | if ${EXEC_COMMANDS} ; then | ||
| + | execute_commands | ||
| + | fi | ||
| - | rm -rf $tmpdir | + | # Finish by system logs to prevent space errors |
| + | echo "Extracting system logs" | ||
| + | if [ "${option}" = "minimal" ]; then | ||
| + | copy_if_exists /var/log/messages ${tmpdir}/var/log/ | ||
| + | copy_if_exists /var/log/dmesg ${tmpdir}/var/log/ | ||
| + | for filename in /var/log/*.1; do | ||
| + | copy_if_exists ${filename} ${tmpdir}/var/log/ | ||
| + | done | ||
| + | for filename in /var/log/*.log; do | ||
| + | copy_if_exists ${filename} ${tmpdir}/var/log/ | ||
| + | done | ||
| + | else | ||
| + | copy_if_exists /var/log/ ${tmpdir}/var/ | ||
| + | fi | ||
| + | |||
| + | echo "Saving all data in ${output_dir}/${name}.tar.gz" | ||
| + | tar -C /user -czf ${output_dir}/${name}.tar.gz ${name} | ||
| + | echo "Done" | ||
| + | sync | ||
| + | |||
| + | rm -rf $tmpdir | ||
| } | } | ||
| Line 950: | Line 1068: | ||
| key="$1" | key="$1" | ||
| - | case $key in | + | case $key in |
| - | -l|--ignore-leds) | + | -l|--ignore-leds) |
| - | LEDS_MANAGEMENT=false | + | LEDS_MANAGEMENT=false |
| - | shift | + | shift |
| - | ;; | + | ;; |
| - | -o|--output-dir) | + | -o|--output-dir) |
| - | OUTPUT_DIR="$2" | + | OUTPUT_DIR="$2" |
| - | shift 2 | + | shift 2 |
| - | ;; | + | ;; |
| - | -c|--disable-commands) | + | -c|--disable-commands) |
| - | EXEC_COMMANDS=false | + | EXEC_COMMANDS=false |
| - | shift | + | shift |
| - | ;; | + | ;; |
| - | -h|--help) | + | -h|--help) |
| - | show_help | + | show_help |
| - | shift | + | shift |
| - | exit 1 | + | exit 1 |
| - | ;; | + | ;; |
| - | -v|--version) | + | -v|--version) |
| - | show_version | + | show_version |
| - | shift | + | shift |
| - | exit 1 | + | exit 1 |
| - | ;; | + | ;; |
| - | *) | + | -m|--minimal) |
| - | shift | + | OPTION="minimal" |
| - | ;; | + | shift |
| - | esac | + | ;; |
| + | *) | ||
| + | shift | ||
| + | ;; | ||
| + | esac | ||
| done | done | ||
| Line 990: | Line 1112: | ||
| # Set led On | # Set led On | ||
| if ${LEDS_MANAGEMENT} ; then | if ${LEDS_MANAGEMENT} ; then | ||
| - | detect_leds | + | detect_leds |
| - | set_led none 1 | + | set_led none 1 |
| fi | fi | ||
| # Get logs | # Get logs | ||
| - | get_logs ${OUTPUT_DIR} | + | get_logs ${OUTPUT_DIR} ${OPTION} |
| # End of script, manage user's notifications | # End of script, manage user's notifications | ||
| if ${LEDS_MANAGEMENT} ; then | if ${LEDS_MANAGEMENT} ; then | ||
| - | # Prepare reboot after USB unplug | + | # Prepare reboot after USB unplug |
| - | cat > /tmp/usbkey_remove << EOF | + | cat > /tmp/usbkey_remove << EOF |
| # Switch off Led | # Switch off Led | ||
| echo "none" > ${LED_PATH}/trigger | echo "none" > ${LED_PATH}/trigger | ||
| EOF | EOF | ||
| - | # All is right, make Led blinking fast to notify User | + | # All is right, make Led blinking fast to notify User |
| - | set_led timer 100 100 | + | set_led timer 100 100 |
| fi | fi | ||
| </code> | </code> | ||
| ++++ | ++++ | ||
| + | |||
| + | ===== Helium ===== | ||
| + | This section contains some tips to do the first level of analysis (support level 1) on Helium gateways.\\ | ||
| + | It will help you to investigate before escalate to Kerlink support (level 2). | ||
| + | |||
| + | |||
| + | **Please note that there is an existing open wiki for Helium users: [[https://wikikerlink.fr/helium|Kerlink Helium Hotspot Wiki]]**\\ | ||
| + | * First step is to follow [[https://wikikerlink.fr/helium/doku.php?id=home#troubleshooting_guide|Troubleshooting guide]] chapter which contains: | ||
| + | * [[https://wikikerlink.fr/helium/doku.php?id=home#self_diagnostic|Self diagnostic]] section | ||
| + | * [[https://wikikerlink.fr/helium/doku.php?id=home#useful_links|Useful links]] section | ||
| + | |||
| + | * Second step, if necessary, is to follow the table below: | ||
| + | |||
| + | ^ Source of info ^ KPI ^ Value ^ Action ^ | ||
| + | ^ [[https://helium-onboarding.kerlink.com/|Onboarding dashboard]] | Last seen | More than 30m ago | Gateway is offline. Ask customer to check power/internet. | | ||
| + | | ::: | ::: | Less than 30m ago | Gateway is online. Check next KPI. | | ||
| + | | ::: | Block height | = 0 | Gateway is downloading a snapshot. | | ||
| + | | ::: | ::: | = 1 | Gateway is loading the downloaded snapshot. | | ||
| + | | ::: | ::: | Else | KPI is ok, check next KPI. | | ||
| + | | ::: | Block age | > 30 mn | Gateway is syncing, check again in one hour. If it persists, you can open a ticket to Kerlink Support. | | ||
| + | | ::: | ::: | < 30 mn | Gateway is synced. Check next KPI. | | ||
| + | ^ [[https://explorer.helium.com//|Helium explorer]] | 24h/14d earnings | > 0 HNT | Seems OK. Confirm with other Explorer KPIs. | | ||
| + | | ::: | Last activity date | < 3h | Seems OK. Confirm with other Explorer KPIs. | | ||
| + | | ::: | Is there a constructed challenge in the past 24h? | Yes | Seems OK. Confirm with other Explorer KPIs. | | ||
| + | | ::: | ::: | No | When gateway is synced, it should challenge. You can open a ticket to Kerlink Support. | | ||
| + | | ::: | Is there a challenged beaconer in the past 24h? | Yes | Seems OK. Confirm with other Explorer KPIs. | | ||
| + | | ::: | ::: | No | Gateway might be relayed. If not, you can open a ticket to Kerlink Support. | | ||
| + | | ::: | Is there a witnessed beacon in the past 8-24h? | Value depends on density around miner | Seems OK. Confirm with other Explorer KPIs. | | ||
| + | | ::: | ::: | No | You can open a ticket to Kerlink Support. | | ||
| + | | ::: | Is there a “broadcasted beacon” in the past 5d? | Yes | Seems OK. Confirm with other Explorer KPIs. | | ||
| + | | ::: | ::: | No | This is random but it seems odd. You can open a ticket to Kerlink Support. | | ||
| + | |||
| + | ==== Gateways connected to a WMC==== | ||
| + | For gateways connected to a WMC, you can use the Remote Command: Remote Access > Command\\ | ||
| + | {{:images:remote_command_wanesy_management_center.png?800|}} | ||
| + | <note important>** Due its cryptographic key and to ensure Kerlink Helium Hotspot security, no SSH access can be given to those gateways. \\ **</note> | ||
| + | |||
| + | You can use commands below: | ||
| + | ^ Command ^ Awaited result ^ | ||
| + | | opkg list | {{:images:opkg.png?400|}} | | ||
| + | | monit status miner | {{:images:monit_status_miner.png?400|}} |Status = OK | | ||
| + | | monit status lorafwd_helium | {{:images:monit_status_lorafwd_helium.png?400|}} | Status = OK | | ||
| + | | /etc/init.d/miner status | {{:images:miner_status.png?400|}} | Check Diff | | ||
| + | |||
| + | If the result is different than the awaited result, you can open a ticket to Kerlink Support. | ||
wiki/support/troubleshoot.1584548176.txt.gz · Last modified: 2020/03/18 17:16 by bdu
