Wirnet™ i-series

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Troubleshoot the gateway ======


===== Useful logs =====

When configuring the gateway or when something unusual happens, most of the time, to understand the behaviour of the gateway it is advised to consult the logs.
Here is the list of the most useful logs:

  * ''/var/log/messages*'': These files are the main logs, they gather the traces of the firmware. They contain the boot traces, the network events, the hardware events, and many other traces.
  * ''/usr/spf*/var/log/spf2.log*'': Theses files contains the traces of the packet forwarder. If you encounter LoRa issues, this is a good way to start.

<note tip>
FIXME : adapt for CPF
</note>

  * ''/tmp/board_info.json'': This file contains information about the hardware of the gateway. It is generated at each boot time.
  * ''/tmp/calib_loraloc.json'': This file contains the calibration values specific to this gateway. It is generated at boot time.
  * ''/dev/nmea*'': These devices receive nmea frames (GPS) every seconds. It can be monitored with the ''cat'' command. 

===== Useful commands =====

Since Kerlink gateways use a Unix system, most current commands are available (ifconfig, route, ps, iptables, ...).

**KerOS specific commands:**
  * ''gsmdiag.py'': generates a status of the modem.
  * ''connmanctl'': used to configure/monitor the network manager.
  * ''opkg list-installed'': lists the installed packages.
  * ''monit status'': displays the status of the monitored processes.

===== Bootcause =====

When a gateway is started/restarted, U-Boot displays the boot cause. The possible reasons are:

  * POR: Power On Reset (happens when the gateway is powered or after a powerloss).
  * WDOG: Watchdog.
  * SW: Software Reset.
  * HW: Physical reset (RESET button).
  * UNKNOWN: unknown cause.

This information is transmitted to initrd and to applications with the kernel argument « bootcause ». \\
The information is then provided in the "bootcause" parameter in the file **''/proc/cmdline''**. \\
Example:
<code bash>
root@klk-lpbs-XXXXXX:~ # cat /proc/cmdline 
console=ttymxc0,115200 bootmode=nominal bootcause=SW bootcount=5 bootfail=0
</code>

The bootcount value is counting the number of reboots since the last power cycle. At each power loss the counter is reset to 0. The maximum value of this counter is 65535.

===== Contacting Kerlink's support team =====

In case you need assistance to troubleshoot your gateway, Kerlink's support team will help you if:
  * The gateway is still under warranty.
  * The gateway has been bought directly from Kerlink.

If the gateway does not satisfy this requirement, please contact [[sales@kerlink.fr]]. Our Sales team will make you an offer to get assistance.

Otherwise, contact [[support@kerlink.fr]] to get assistance. Our support team will open you an account in Kerlink's ticketing tool.

When contacting the support team, to improve and speed up the resolution of your issue, please provide them:
  * The serial number of your gateway(s) (the serial number is written on a sticker inside the case with "Wirnet iBTS" written on it or the "Final product ID" on the rear sticker for Wirnet iFemtoCell).
  * A precise description of your problem. "Gateway not working" is not enough to investigate.
  * The list of the actions you took to troubleshoot/investigate your problem.
  * The date of your problem.
  * The version of your firmware.
  * The logs of your gateway (refer to the automatic log gathering section). Without these logs, it is difficult to investigate. If you are able to reproduce the problem, make sure to reproduce it and gather the logs afterward.
  * Any information that would help us understand the context. For example, if the gateway is on the field or in a lab, if a third party program is used, if the gateway was previously working, etc.

===== Automatic log gathering =====

To simplify the log gathering process, two scripts are available. These scripts generate an archive containing many logs and commands results.

<note tip>
TODO 
  * adapt for CPF logs
  * add result of overlay diff
  * add mount command
  * remove reference to /user/rootfs_rw
</note>

++++Gathering logs from shell|

To gather the logs from shell:
  * Upload the script bellow (use whatever method you prefer: FileZilla, scp, copy/paste, ...) in a folder of your gateway and go to this folder.
  * Convert the file to Unix text file format using ''dos2unix gather_logs.sh''.
  * Add execution rights to the script using ''chmod 755 gather_logs.sh''.
  * Execute the script using ''./gather_logs.sh''. //The execution of the script takes about 1-2 minutes.//
  * It will generate an archive under ''/tmp''.
  * Retrieve this archive (use whatever method you prefer: FileZilla, scp, ...).

<code bash gather_logs.sh>
#!/bin/sh

# get the name and the MAC of the board
reg=$(cat /sys/fsl_otp/HW_OCOTP_GP1)
serial=$(printf "%X\n" $reg)
shortmac=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}')

NAME="Logs_${serial}_${shortmac}_$(date "+%Y%m%d-%H%M%S")"
tmpdir="/tmp/${NAME}"

# generate a temporary directory
mkdir -p ${tmpdir}/

{
	# get general information
	{ uname -a; echo; date; echo; uptime; } > ${tmpdir}/status.txt
	ps ww > ${tmpdir}/ps.log

	# get /var/log logs
	mkdir -p ${tmpdir}/var/log/
	cp -a /var/log/* ${tmpdir}/var/log/

	# get packet forwarder logs/configuration
	shopt -s extglob
	mkdir -p ${tmpdir}/packet_forwarder/
	cp -a /user/!(rootfs_rw)/var/log/* ${tmpdir}/packet_forwarder/
	cp -a /user/!(rootfs_rw)/etc/* ${tmpdir}/packet_forwarder/		

	# get /tmp files
	cp /tmp/sys_startup_status.json ${tmpdir}/
	cp /tmp/calib_loraloc.json ${tmpdir}/
	cp /tmp/gps_vers.txt ${tmpdir}/

	# get user configuration
	mkdir -p ${tmpdir}/config/
	cp -a /user/rootfs_rw/etc/* ${tmpdir}/config/

	# get network information
	ifconfig -a > ${tmpdir}/ifconfig.log
	route > ${tmpdir}/route.log
	iptables -L > ${tmpdir}/iptables.log
	netstat > ${tmpdir}/netstat.log
	{ echo 'connmanctl technologies'
	  connmanctl technologies

	  echo
	  echo 'connmanctl services'
	  connmanctl services
	} > ${tmpdir}/connmanctl.log

	# get GSM information
	gsmdiag.py ; cp /tmp/gsmdiag.txt ${tmpdir}/
	{ echo 'qmicli -d /dev/cdc-wdm0 --dms-get-ids'
	  qmicli -d /dev/cdc-wdm0 --dms-get-ids

	  echo 	
	  echo 'qmicli -d /dev/cdc-wdm0 --nas-get-serving-system'
	  qmicli -d /dev/cdc-wdm0 --nas-get-serving-system

	  echo 	
	  echo 'qmicli -d /dev/cdc-wdm0 --nas-get-system-selection-preference'
	  qmicli -d /dev/cdc-wdm0 --nas-get-system-selection-preference
	  
	  echo 	
	  echo 'qmicli -d /dev/cdc-wdm0 --dms-get-operating-mode'
	  qmicli -d /dev/cdc-wdm0 --dms-get-operating-mode
	} > ${tmpdir}/qmicli_0.log

	# get GPS info
	timeout -t 10 cat /dev/nmea2 > /dev/null
	head -n 50 /dev/nmea2 > ${tmpdir}/nmea2.log

	# get LoRa information
	loraloc_info > ${tmpdir}/loraloc_info.log

	# get miscellaneous information
	find /user/ > ${tmpdir}/user_file_list.txt
	opkg list-installed > ${tmpdir}/installed-packages.log	
        monit status > ${tmpdir}/monit_status.log	
} > ${tmpdir}/gather_logs.log 2>&1

tar -C /tmp -czf /tmp/${NAME}.tar.gz ${NAME}
echo "/tmp/${NAME}.tar.gz created"
</code>

++++

++++Gathering logs from USB (no commands required)|

To gather logs from a USB without typing any command:
  * Download the following script and copy it on a USB key.
  * Create a ''usbkey.txt'' file on the USB key as described [[wiki:sw_updates#specific_files|here]].
  * Plug the USB key on the CPU board.
  * Wait for the yellow debug led fast blinking. //The execution of the script takes about 1-2 minutes.//
  * Unplug the USB key.
  * An archive with all the required logs has been created on the USB key.

<code bash usb.autorun>
#!/bin/sh

USBDIR="`dirname $0`"

SYSLEDPATH_IBTS="/sys/class/leds/yellow:debug/"
SYSLEDPATH_IFEMTO="/sys/class/leds/led1:red:power/"

HW_IBTS="0x2e"
HW_IFEMTO="0x39 0x44 0x45"

#
# Check platform we are running on
#
get_platform() {
	for hwversion in ${HW_IBTS}
	do
		if [ "$(cat /sys/fsl_otp/HW_OCOTP_GP1 | grep ${hwversion})" != "" ]; then
				platform="IBTS"
				echo ${platform}
				return
		fi
	done
	for hwversion in ${HW_IFEMTO}
	do
		if [ "$(cat /sys/fsl_otp/HW_OCOTP_GP1 | grep ${hwversion})" != "" ]; then
				platform="IFEMTO"
				echo ${platform}
				return
		fi
	done
	echo "Hardware platform not supported with code : $(cat /var/tmp/version)"
	platform="NOT_SUPPORTED"
	echo ${platform}
}


#
# Set default led behavior
# $1: trigger
#
#    if trigger = timer : $2 = delay_on, $3 = delay_off
#
set_led() {
	trigger="${1}"
	on="$2"
	off="$3"

	if [ -d "${SYSLEDPATH}" ]; then
		echo $trigger > "${SYSLEDPATH}/trigger"
		case "$trigger" in
			"timer")
				echo ${on:=500} > "${SYSLEDPATH}/delay_on"
				echo ${off:=500} > "${SYSLEDPATH}/delay_off"
				;;
			"none")
				echo ${on:=0} > "${SYSLEDPATH}/brightness"
				;;
			*)
				# Nothing to do
				;;
		esac
	fi
}


get_logs()
{
	NAME="Logs_${serial}_${shortmac}_$(date "+%Y%m%d-%H%M%S")"
	tmpdir="/tmp/${NAME}"

	mkdir -p ${tmpdir}/

	{
		# get general information
		{ uname -a; echo; date; echo; uptime; } > ${tmpdir}/status.txt
		ps ww > ${tmpdir}/ps.log

		# get /var/log logs
		mkdir -p ${tmpdir}/var/log/
		cp -a /var/log/* ${tmpdir}/var/log/

		# get packet forwarder logs/configuration
		shopt -s extglob
		mkdir -p ${tmpdir}/packet_forwarder/
		cp -a /user/!(rootfs_rw)/var/log/* ${tmpdir}/packet_forwarder/
		cp -a /user/!(rootfs_rw)/etc/* ${tmpdir}/packet_forwarder/

		# get /tmp files
		cp /tmp/sys_startup_status.json ${tmpdir}/
		cp /tmp/calib_loraloc.json ${tmpdir}/
		cp /tmp/gps_vers.txt ${tmpdir}/
		
		# get user configuration
		mkdir -p ${tmpdir}/config/
		cp -a /user/rootfs_rw/etc/* ${tmpdir}/config/

		# get network information
		ifconfig -a > ${tmpdir}/ifconfig.log
		route > ${tmpdir}/route.log
		iptables -L > ${tmpdir}/iptables.log
		netstat > ${tmpdir}/netstat.log
		{ echo 'connmanctl technologies'
		  connmanctl technologies 

		  echo
		  echo 'connmanctl services'
	  	  connmanctl services 
		} > ${tmpdir}/connmanctl.log
		
		# get GSM information
		gsmdiag.py ; cp /tmp/gsmdiag.txt ${tmpdir}/
		{ echo 'qmicli -d /dev/cdc-wdm0 --dms-get-ids'
		  qmicli -d /dev/cdc-wdm0 --dms-get-ids

		  echo 	
		  echo 'qmicli -d /dev/cdc-wdm0 --nas-get-serving-system'
		  qmicli -d /dev/cdc-wdm0 --nas-get-serving-system

		  echo 	
		  echo 'qmicli -d /dev/cdc-wdm0 --nas-get-system-selection-preference'
		  qmicli -d /dev/cdc-wdm0 --nas-get-system-selection-preference
		  
		  echo 	
		  echo 'qmicli -d /dev/cdc-wdm0 --dms-get-operating-mode'
		  qmicli -d /dev/cdc-wdm0 --dms-get-operating-mode
		} > ${tmpdir}/qmicli_0.log

		# get GPS info
		timeout -t 10 cat /dev/nmea2 > /dev/null
		head -n 50 /dev/nmea2 > ${tmpdir}/nmea2.log

		# get LoRa information
		loraloc_info > ${tmpdir}/loraloc_info.log

		cp /user/rootfs_rw/orbiwise/gwdaemon.log.?.gz ${tmpdir}/orbiwise/

		# get miscellaneous information
		find /user/ > ${tmpdir}/user_file_list.txt
		opkg list-installed > ${tmpdir}/installed-packages.log	
                monit status > ${tmpdir}/monit_status.log	
	} > ${tmpdir}/gather_logs.log 2>&1

	tar -C /tmp -czf ${1}/${NAME}.tar.gz ${NAME}
	echo "${1}/${NAME}.tar.gz created"
	sync

	rm -rf $tmpdir
}


#####################
#     MAIN
#####################

platform=$(get_platform)

reg=$(cat /sys/fsl_otp/HW_OCOTP_GP1)
serial=$(printf "%X\n" $reg)
shortmac=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}')

if [ ${platform} == "IBTS" ]; then
	SYSLEDPATH=${SYSLEDPATH_IBTS}
elif [ ${platform} == "IFEMTO" ]; then
	SYSLEDPATH=${SYSLEDPATH_IFEMTO}
fi

# Set led On
set_led none 1

# Get logs
get_logs ${USBDIR}

# Prepare reboot after USB unplug
cat > /tmp/usbkey_remove << EOF

# Switch off Led
echo "none" > ${SYSLEDPATH}/trigger
EOF

# All is right, make Led blinking fast to notify User
set_led timer 100 100
</code>

++++
Wirnet™ i-series

User Tools

Site Tools

Page Tools
