Differences

This shows you the differences between two versions of the page.

--- wiki:troubleshoot [2019/01/08 15:44]
hch [Bootcause]
+++ — (current)
@@ Line 1: / Line 1: @@
-====== Troubleshoot the gateway ======
-===== Useful logs =====
-When configuring the gateway or when something unusual happens, most of the time, to understand the behaviour of the gateway it is advised to consult the logs.
-Here is the list of the most useful logs:
-  * ''/var/log/messages*'': These files are the main logs, they gather the traces of the firmware. They contain the boot traces, the network events, the hardware events, and many other traces.
-  * ''/usr/spf*/var/log/spf2.log*'': Theses files contains the traces of the packet forwarder. If you encounter LoRa issues, this is a good way to start.
-<note tip>
-FIXME : adapt for CPF
-</note>
-  * ''/tmp/board_info.json'': This file contains information about the hardware of the gateway. It is generated at each boot time.\\
-FIXME
-  * ''/tmp/calib_loraloc.json'': This file contains the calibration values specific to this gateway. It is generated at boot time.
-  * ''/dev/nmea*'' (**Wirnet iBTS only**): These devices receive nmea frames (GPS) every seconds. It can be monitored with the ''cat'' command.
-===== Useful commands =====
-Since Kerlink gateways use a Unix system, most current commands are available (ifconfig, route, ps, iptables, ...).
-**KerOS specific commands:**
-  * ''gsmdiag.py'': generates a status of the modem.
-  * ''connmanctl'': used to configure/monitor the network manager.
-  * ''opkg list-installed'': lists the installed packages.
-  * ''monit status'': displays the status of the monitored processes.
-===== Bootcause =====
-The ''bootcount'' value is counting the number of reboots since the last power cycle.
-At each power loss the counter is reset to 0. The maximum value of this counter is 65535.\\
-The ''bootcount'' value can be checked in the file ''/tmp/board_info.json'':
-<code bash>
-# cat /tmp/board_info.json | grep bootcount
-        "bootcount": 1,
-</code>
-When a gateway is started/restarted, U-Boot displays the boot cause.\\
-The possible reasons are:
-  * POR: Power On Reset (happens when the gateway is powered or after a powerloss).
-  * WDOG: Watchdog.
-  * SW: Software Reset.
-  * HW: Physical reset (RESET button).
-  * UNKNOWN: unknown cause.
-This information is transmitted to initrd and to applications with the kernel argument « bootcause ». \\
-The information is then provided in the ''bootcause'' parameter in the file **''/proc/cmdline''**. \\
-Example:
-<code bash>
-# cat /proc/cmdline
-console=ttymxc0,115200 bootmode=nominal bootcause=SW bootcount=5 bootfail=0
-</code>
-===== Contacting Kerlink's support team =====
-In case you need assistance to troubleshoot your gateway, Kerlink's support team will help you if:
-  * The gateway is still under warranty.
-  * The gateway has been bought directly from Kerlink.
-If the gateway does not satisfy this requirement, please contact [[sales@kerlink.fr]]. Our Sales team will make you an offer to get assistance.
-Otherwise, contact [[support@kerlink.fr]] to get assistance. Our support team will open you an account in Kerlink's ticketing tool.
-When contacting the support team, to improve and speed up the resolution of your issue, please provide them:
-  * The serial number of your gateway(s) (the serial number is written on a sticker inside the case with "Wirnet iBTS" written on it or the "Final product ID" on the rear sticker for Wirnet iFemtoCell).
-  * A precise description of your problem. "Gateway not working" is not enough to investigate.
-  * The list of the actions you took to troubleshoot/investigate your problem.
-  * The date of your problem.
-  * The version of your firmware.
-  * The logs of your gateway (refer to the automatic log gathering section). Without these logs, it is difficult to investigate. If you are able to reproduce the problem, make sure to reproduce it and gather the logs afterward.
-  * Any information that would help us understand the context. For example, if the gateway is on the field or in a lab, if a third party program is used, if the gateway was previously working, etc.
-===== Automatic log gathering =====
-To simplify the log gathering process, two scripts are available. These scripts generate an archive containing many logs and commands results.
-<note tip>
-TODO
-  * adapt for CPF logs
-  * add result of overlay diff
-  * add mount command
-  * remove reference to /user/rootfs_rw
-</note>
-++++Gathering logs from shell|
-To gather the logs from shell:
-  * Upload the script bellow (use whatever method you prefer: FileZilla, scp, copy/paste, ...) in a folder of your gateway and go to this folder.
-  * Convert the file to Unix text file format using ''dos2unix gather_logs.sh''.
-  * Add execution rights to the script using ''chmod 755 gather_logs.sh''.
-  * Execute the script using ''./gather_logs.sh''. //The execution of the script takes about 1-2 minutes.//
-  * It will generate an archive under ''/tmp''.
-  * Retrieve this archive (use whatever method you prefer: FileZilla, scp, ...).
-<code bash gather_logs.sh>
-#!/bin/sh
-# get the name and the MAC of the board
-reg=$(cat /sys/fsl_otp/HW_OCOTP_GP1)
-serial=$(printf "%X\n" $reg)
-shortmac=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}')
-NAME="Logs_${serial}_${shortmac}_$(date "+%Y%m%d-%H%M%S")"
-tmpdir="/tmp/${NAME}"
-# generate a temporary directory
-mkdir -p ${tmpdir}/
-{
-	# get general information
-	{ uname -a; echo; date; echo; uptime; } > ${tmpdir}/status.txt
-	ps ww > ${tmpdir}/ps.log
-	# get /var/log logs
-	mkdir -p ${tmpdir}/var/log/
-	cp -a /var/log/* ${tmpdir}/var/log/
-	# get packet forwarder logs/configuration
-	shopt -s extglob
-	mkdir -p ${tmpdir}/packet_forwarder/
-	cp -a /user/!(rootfs_rw)/var/log/* ${tmpdir}/packet_forwarder/
-	cp -a /user/!(rootfs_rw)/etc/* ${tmpdir}/packet_forwarder/
-	# get /tmp files
-	cp /tmp/sys_startup_status.json ${tmpdir}/
-	cp /tmp/calib_loraloc.json ${tmpdir}/
-	cp /tmp/gps_vers.txt ${tmpdir}/
-	# get user configuration
-	mkdir -p ${tmpdir}/config/
-	cp -a /user/rootfs_rw/etc/* ${tmpdir}/config/
-	# get network information
-	ifconfig -a > ${tmpdir}/ifconfig.log
-	route > ${tmpdir}/route.log
-	iptables -L > ${tmpdir}/iptables.log
-	netstat > ${tmpdir}/netstat.log
-	{ echo 'connmanctl technologies'
-	  connmanctl technologies
-	  echo
-	  echo 'connmanctl services'
-	  connmanctl services
-	} > ${tmpdir}/connmanctl.log
-	# get GSM information
-	gsmdiag.py ; cp /tmp/gsmdiag.txt ${tmpdir}/
-	{ echo 'qmicli -d /dev/cdc-wdm0 --dms-get-ids'
-	  qmicli -d /dev/cdc-wdm0 --dms-get-ids
-	  echo
-	  echo 'qmicli -d /dev/cdc-wdm0 --nas-get-serving-system'
-	  qmicli -d /dev/cdc-wdm0 --nas-get-serving-system
-	  echo
-	  echo 'qmicli -d /dev/cdc-wdm0 --nas-get-system-selection-preference'
-	  qmicli -d /dev/cdc-wdm0 --nas-get-system-selection-preference
-	  echo
-	  echo 'qmicli -d /dev/cdc-wdm0 --dms-get-operating-mode'
-	  qmicli -d /dev/cdc-wdm0 --dms-get-operating-mode
-	} > ${tmpdir}/qmicli_0.log
-	# get GPS info
-	timeout -t 10 cat /dev/nmea2 > /dev/null
-	head -n 50 /dev/nmea2 > ${tmpdir}/nmea2.log
-	# get LoRa information
-	loraloc_info > ${tmpdir}/loraloc_info.log
-	# get miscellaneous information
-	find /user/ > ${tmpdir}/user_file_list.txt
-	opkg list-installed > ${tmpdir}/installed-packages.log
-        monit status > ${tmpdir}/monit_status.log
-} > ${tmpdir}/gather_logs.log 2>&1
-tar -C /tmp -czf /tmp/${NAME}.tar.gz ${NAME}
-echo "/tmp/${NAME}.tar.gz created"
-</code>
-++++
-++++Gathering logs from USB (no commands required)|
-To gather logs from a USB without typing any command:
-  * Download the following script and copy it on a USB key.
-  * Create a ''usbkey.txt'' file on the USB key as described [[wiki:sw_updates#specific_files|here]].
-  * Plug the USB key on the CPU board.
-  * Wait for the yellow debug led fast blinking. //The execution of the script takes about 1-2 minutes.//
-  * Unplug the USB key.
-  * An archive with all the required logs has been created on the USB key.
-<code bash usb.autorun>
-#!/bin/sh
-USBDIR="`dirname $0`"
-SYSLEDPATH_IBTS="/sys/class/leds/yellow:debug/"
-SYSLEDPATH_IFEMTO="/sys/class/leds/led1:red:power/"
-HW_IBTS="0x2e"
-HW_IFEMTO="0x39 0x44 0x45"
-#
-# Check platform we are running on
-#
-get_platform() {
-	for hwversion in ${HW_IBTS}
-	do
-		if [ "$(cat /sys/fsl_otp/HW_OCOTP_GP1 | grep ${hwversion})" != "" ]; then
-				platform="IBTS"
-				echo ${platform}
-				return
-		fi
-	done
-	for hwversion in ${HW_IFEMTO}
-	do
-		if [ "$(cat /sys/fsl_otp/HW_OCOTP_GP1 | grep ${hwversion})" != "" ]; then
-				platform="IFEMTO"
-				echo ${platform}
-				return
-		fi
-	done
-	echo "Hardware platform not supported with code : $(cat /var/tmp/version)"
-	platform="NOT_SUPPORTED"
-	echo ${platform}
-}
-#
-# Set default led behavior
-# $1: trigger
-#
-#    if trigger = timer : $2 = delay_on, $3 = delay_off
-#
-set_led() {
-	trigger="${1}"
-	on="$2"
-	off="$3"
-	if [ -d "${SYSLEDPATH}" ]; then
-		echo $trigger > "${SYSLEDPATH}/trigger"
-		case "$trigger" in
-			"timer")
-				echo ${on:=500} > "${SYSLEDPATH}/delay_on"
-				echo ${off:=500} > "${SYSLEDPATH}/delay_off"
-				;;
-			"none")
-				echo ${on:=0} > "${SYSLEDPATH}/brightness"
-				;;
-			*)
-				# Nothing to do
-				;;
-		esac
-	fi
-}
-get_logs()
-{
-	NAME="Logs_${serial}_${shortmac}_$(date "+%Y%m%d-%H%M%S")"
-	tmpdir="/tmp/${NAME}"
-	mkdir -p ${tmpdir}/
-	{
-		# get general information
-		{ uname -a; echo; date; echo; uptime; } > ${tmpdir}/status.txt
-		ps ww > ${tmpdir}/ps.log
-		# get /var/log logs
-		mkdir -p ${tmpdir}/var/log/
-		cp -a /var/log/* ${tmpdir}/var/log/
-		# get packet forwarder logs/configuration
-		shopt -s extglob
-		mkdir -p ${tmpdir}/packet_forwarder/
-		cp -a /user/!(rootfs_rw)/var/log/* ${tmpdir}/packet_forwarder/
-		cp -a /user/!(rootfs_rw)/etc/* ${tmpdir}/packet_forwarder/
-		# get /tmp files
-		cp /tmp/sys_startup_status.json ${tmpdir}/
-		cp /tmp/calib_loraloc.json ${tmpdir}/
-		cp /tmp/gps_vers.txt ${tmpdir}/
-		# get user configuration
-		mkdir -p ${tmpdir}/config/
-		cp -a /user/rootfs_rw/etc/* ${tmpdir}/config/
-		# get network information
-		ifconfig -a > ${tmpdir}/ifconfig.log
-		route > ${tmpdir}/route.log
-		iptables -L > ${tmpdir}/iptables.log
-		netstat > ${tmpdir}/netstat.log
-		{ echo 'connmanctl technologies'
-		  connmanctl technologies
-		  echo
-		  echo 'connmanctl services'
-	  	  connmanctl services
-		} > ${tmpdir}/connmanctl.log
-		# get GSM information
-		gsmdiag.py ; cp /tmp/gsmdiag.txt ${tmpdir}/
-		{ echo 'qmicli -d /dev/cdc-wdm0 --dms-get-ids'
-		  qmicli -d /dev/cdc-wdm0 --dms-get-ids
-		  echo
-		  echo 'qmicli -d /dev/cdc-wdm0 --nas-get-serving-system'
-		  qmicli -d /dev/cdc-wdm0 --nas-get-serving-system
-		  echo
-		  echo 'qmicli -d /dev/cdc-wdm0 --nas-get-system-selection-preference'
-		  qmicli -d /dev/cdc-wdm0 --nas-get-system-selection-preference
-		  echo
-		  echo 'qmicli -d /dev/cdc-wdm0 --dms-get-operating-mode'
-		  qmicli -d /dev/cdc-wdm0 --dms-get-operating-mode
-		} > ${tmpdir}/qmicli_0.log
-		# get GPS info
-		timeout -t 10 cat /dev/nmea2 > /dev/null
-		head -n 50 /dev/nmea2 > ${tmpdir}/nmea2.log
-		# get LoRa information
-		loraloc_info > ${tmpdir}/loraloc_info.log
-		cp /user/rootfs_rw/orbiwise/gwdaemon.log.?.gz ${tmpdir}/orbiwise/
-		# get miscellaneous information
-		find /user/ > ${tmpdir}/user_file_list.txt
-		opkg list-installed > ${tmpdir}/installed-packages.log
-                monit status > ${tmpdir}/monit_status.log
-	} > ${tmpdir}/gather_logs.log 2>&1
-	tar -C /tmp -czf ${1}/${NAME}.tar.gz ${NAME}
-	echo "${1}/${NAME}.tar.gz created"
-	sync
-	rm -rf $tmpdir
-}
-#####################
-#     MAIN
-#####################
-platform=$(get_platform)
-reg=$(cat /sys/fsl_otp/HW_OCOTP_GP1)
-serial=$(printf "%X\n" $reg)
-shortmac=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}')
-if [ ${platform} == "IBTS" ]; then
-	SYSLEDPATH=${SYSLEDPATH_IBTS}
-elif [ ${platform} == "IFEMTO" ]; then
-	SYSLEDPATH=${SYSLEDPATH_IFEMTO}
-fi
-# Set led On
-set_led none 1
-# Get logs
-get_logs ${USBDIR}
-# Prepare reboot after USB unplug
-cat > /tmp/usbkey_remove << EOF
-# Switch off Led
-echo "none" > ${SYSLEDPATH}/trigger
-EOF
-# All is right, make Led blinking fast to notify User
-set_led timer 100 100
-</code>
-++++

Wirnet™ i-series

User Tools

Site Tools

Differences

Page Tools