This is an old revision of the document!

Troubleshoot the gateway

Useful logs

When configuring the gateway or when something unusual happens, most of the time, to understand the behaviour of the gateway it is advised to consult the logs. Here is the list of the most useful logs:

/var/log/messages*: These files are the main logs, they gather the traces of the firmware. They contain the boot traces, the network events, the hardware events, and many other traces.
/var/log/lora.log*: Theses files contains the traces of the common packet forwarder (lorad and lorafwd). If you encounter LoRa issues, this is a good way to start. The verbosity of this log can be increased/decreased. Refer to the log management section.
/tmp/board_info.json: This file contains information about the hardware of the gateway. It is generated at each boot time.
/tmp/calib_loraloc.json (for Wirnet iBTS): This file contains the calibration values specific to this gateway. It is generated at boot time.
/tmp/calib_rf.json (for Wirnet iFemtoCell and Wirnet iStation): This file contains the calibration values specific to this gateway. It is generated at boot time.
/dev/nmea* (Wirnet iBTS and Wirnet iStation): These devices receive NMEA frames (GPS) every seconds. It can be monitored with the cat command.

Useful commands

Since Kerlink gateways use a Unix system, most current commands are available (ifconfig, route, ps, iptables, …).

KerOS specific commands:

gsmdiag.py: generates a status of the modem.
connmanctl: used to configure/monitor the network manager.
opkg list-installed: lists the installed packages.
monit status: displays the status of the monitored processes.

Bootcause

The bootcount value is counting the number of reboots since the last power cycle. At each power loss the counter is reset to 0. The maximum value of this counter is 65535.
The bootcount value can be checked in the file /tmp/board_info.json:

# grep bootcount /tmp/board_info.json
        "bootcount": 1,

When a gateway is started/restarted, U-Boot displays the boot cause.
The possible reasons are:

POR: Power On Reset (happens when the gateway is powered or after a powerloss).
WDOG: Watchdog.
SW: Software Reset.
HW: Physical reset (RESET button).
UNKNOWN: unknown cause.

This information is transmitted to initrd and to applications with the kernel argument « bootcause ».
The information is then provided in the bootcause parameter in the file /proc/cmdline.
Example:

# cat /proc/cmdline 
console=ttymxc0,115200 bootmode=nominal bootcause=SW bootcount=5 bootfail=0

Contacting Kerlink's support team

In case you need assistance to troubleshoot your gateway, Kerlink's support team will help you if:

The gateway is still under warranty.
The gateway has been bought directly from Kerlink.

If the gateway does not satisfy this requirement, please contact sales@kerlink.fr. Our Sales team will make you an offer to get assistance.

Otherwise, contact support@kerlink.fr to get assistance. Our support team will open you an account in Kerlink's ticketing tool.

When contacting the support team, to improve and speed up the resolution of your issue, please provide them:

The serial number of your gateway(s):
- For Wirnet iBTS, the serial number is written on a sticker inside the case with “Wirnet iBTS” written on it.
- For Wirnet iFemtoCell, the serial number is written on the rear sticker (it is named “Final product ID”).
- For Wirnet iStation, the serial number is written on the external sticker (it is named “Product ID”).
A precise description of your problem. “Gateway not working” is not enough to investigate.
The list of the actions you took to troubleshoot/investigate your problem.
The date of your problem.
The version of your firmware.
The logs of your gateway (refer to the automatic log gathering section). Without these logs, it is difficult to investigate. If you are able to reproduce the problem, make sure to reproduce it and gather the logs afterwards.
Any information that would help us understand the context. For example, if the gateway is on the field or in a lab, if a third party program is used, if the gateway was previously working, etc.

Automatic log gathering

To simplify the log gathering process, two scripts are available. These scripts generate an archive containing many logs and commands results.

Gathering logs from shell

To gather the logs from shell:

Upload the script bellow (use whatever method you prefer: FileZilla, scp, copy/paste, …) in a folder of your gateway and go to this folder.
Convert the file to Unix text file format using dos2unix gather_logs.sh.
Add execution rights to the script using chmod 755 gather_logs.sh.
Execute the script using ./gather_logs.sh. The execution of the script takes about 1-2 minutes.
It will generate an archive under /tmp.
Retrieve this archive (use whatever method you prefer: FileZilla, scp, …).

gather_logs.sh

#!/bin/sh
 
USBDIR="`dirname $0`"
 
SYSLEDPATH_IBTS="/sys/class/leds/yellow:debug/"
SYSLEDPATH_IFEMTO="/sys/class/leds/led1:red:power/"
 
check_version(){
	# check if /etc/version exists
	if [ -a /etc/version ]; then
	    FW_VERSION=$(cat /etc/version)
 
	    # Check if the firmware version is superior or equal to 4. If not, stop the script.
	    if [ $(cut -c 1 <<< $FW_VERSION) -lt 4 ]; then
	        echo "Your firmware version ($FW_VERSION) is inferior to v4. Please use the adequate script."
	        exit
	    fi
	else
	    echo "could not find /etc/version"
	    exit
	fi
}
 
#
# Check platform we are running on
#
get_platform() {
	platform="$(fslotpreg getplatform)"
	if [ $? -ne 0 ];
	then
		platform="NOT_SUPPORTED"
	fi
	echo ${platform}
}
 
 
get_logs()
{
	NAME="Logs_${serial}_${shortmac}_$(date "+%Y-%m-%d_%H-%M-%S")"
	tmpdir="/tmp/${NAME}"
 
	mkdir -p ${tmpdir}/
 
	# get logs
	mkdir -p ${tmpdir}/var/log
	cp -a /var/log/* ${tmpdir}/var/log/
 
	# get tmp files 
	mkdir -p ${tmpdir}/tmp
	cp -a /tmp/sys_startup_status.json ${tmpdir}/tmp/
	if [ ${platform} == "IBTS" ]; then
		cp -a /tmp/calib_loraloc.json ${tmpdir}/tmp/
	elif [ ${platform} == "IFEMTO" ]; then
		cp -a /tmp/calib_rf.json ${tmpdir}/tmp/
	fi
 
	# get network config
	mkdir -p ${tmpdir}/etc/network
	mkdir -p ${tmpdir}/etc/firewall.d
	cp -a /etc/network/* ${tmpdir}/etc/network/
	cp -a /etc/firewall.d/* ${tmpdir}/etc/firewall.d/
 
	# CPF config
	mkdir -p ${tmpdir}/etc/default
	mkdir -p ${tmpdir}/user/etc/lorad
	mkdir -p ${tmpdir}/user/etc/lorafwd
	cp -a /etc/default/lorad ${tmpdir}/etc/default/
	cp -a /etc/default/lorafwd ${tmpdir}/etc/default/
	cp -a /user/etc/lorad/* ${tmpdir}/user/etc/lorad/
	cp -a /user/etc/lorafwd/* ${tmpdir}/user/etc/lorafwd/
 
	# miscaellenous config
	mkdir -p ${tmpdir}/etc/rcU.d
	cp -a /etc/rcU.d/* ${tmpdir}/etc/rcU.d
	cp -a /user/.update ${tmpdir}/user/.update 
 
	(
		echo -e "\n\n\n**************  date  *****************"
		date
		echo -e "\n\n\n**************  uptime  *****************"
		uptime
		echo -e "\n\n\n**************  cat /etc/version  *****************"
		cat /etc/version
		echo -e "\n\n\n**************  ps w  *****************"
		ps w
		echo -e "\n\n\n**************  opkg list-installed  *****************"
		opkg list-installed
		echo -e "\n\n\n**************  opkg list  *****************"
		opkg list		
	 	echo -e "\n\n\n**************  monit status  *****************"
	 	monit status
	 	echo -e "\n\n\n**************  mount  *****************"
	 	mount
	) >> ${tmpdir}/cmd_results.txt
 
	(
		echo -e "\n\n\n**************  ifconfig -a  *****************"
		ifconfig -a
		echo -e "\n\n\n**************  gsmdiag.py  *****************"
		gsmdiag.py ; cat /tmp/gsmdiag.txt
		echo -e "\n\n\n**************  route  *****************"
		route
		echo -e "\n\n\n**************  iptables -L  *****************"
		iptables -L
		echo -e "\n\n\n**************  connmanctl technologies  *****************"
		connmanctl technologies
		echo -e "\n\n\n**************  services  *****************"
		connmanctl services
 
		if [ ${platform} == "IBTS" ]; then
			echo -e "\n\n\n**************  qmicli get mode  *****************"
			qmicli -d /dev/cdc-wdm1 --nas-get-system-selection-preference
		fi
	) >> ${tmpdir}/network_result.txt
 
    # get the 25 last NMEA frames; it will take 10s + 25s
	if [ ${platform} == "IBTS" ]; then
 
	      if [ -n "$(lsof | grep $(readlink /dev/nmea2))" ]; then
		      echo " /dev/nmea2 already used by $(lsof | grep $(readlink /dev/nmea2))" > ${tmpdir}/nmea2.log; 
 
	      else
		      timeout -t 10 cat /dev/nmea2 > /dev/null
		      timeout -t 40 head -n 50 /dev/nmea2 > ${tmpdir}/nmea2.log
              fi
	fi
 
	# create an archive with all the files and command results
	# the archive is created on the USB key
	tar -C /tmp -czf ${NAME}.tar.gz ${NAME}
	echo "Archive ${NAME}.tar.gz created"
	sync
 
	rm -rf $tmpdir
}
 
 
#####################
#     MAIN
#####################
 
check_version
 
platform=$(get_platform)
 
reg=$(fslotpreg read GP1)
if [ $? -eq 0 ]; then
	serial=$(printf "%X\n" $reg)
else
	serial="unknown"
fi
shortmac=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}')
 
# Get logs
get_logs

Gathering logs from USB (no commands required)

To gather logs from a USB without typing any command:

Download the following script and copy it on a USB key.
Create a usbkey.txt file on the USB key as described here.
Plug the USB key on the CPU board.
Wait for the yellow debug led fast blinking. The execution of the script takes about 1-2 minutes.
Unplug the USB key.
An archive with all the required logs has been created on the USB key.

usb.autorun

#!/bin/sh
 
USBDIR="`dirname $0`"
 
SYSLEDPATH_IBTS="/sys/class/leds/yellow:debug/"
SYSLEDPATH_IFEMTO="/sys/class/leds/led1:red:power/"
 
 
check_version(){
	# check if /etc/version exists
	if [ -a /etc/version ]; then
	    FW_VERSION=$(cat /etc/version)
 
	    # Check if the firmware version is superior or equal to 4. If not, stop the script.
	    if [ $(cut -c 1 <<< $FW_VERSION) -lt 4 ]; then
	        echo "Your firmware version ($FW_VERSION) is inferior to v4. Please use the adequate script." >> ${USBDIR}/WRONG_FIRMWARE_VERSION.txt
	        exit
	    fi
	else
	    echo "Could not find /etc/version" >> ${USBDIR}/WRONG_FIRMWARE_VERSION.txt
	    exit
	fi
}
 
 
#
# Check platform we are running on
#
get_platform() {
	platform="$(fslotpreg getplatform)"
	if [ $? -ne 0 ];
	then
		platform="NOT_SUPPORTED"
	fi
	echo ${platform}
}
 
 
#
# Set default led behavior
# $1: trigger
#
#    if trigger = timer : $2 = delay_on, $3 = delay_off
#
set_led() {
	trigger="${1}"
	on="$2"
	off="$3"
 
	if [ -d "${SYSLEDPATH}" ]; then
		echo $trigger > "${SYSLEDPATH}/trigger"
		case "$trigger" in
			"timer")
				echo ${on:=500} > "${SYSLEDPATH}/delay_on"
				echo ${off:=500} > "${SYSLEDPATH}/delay_off"
				;;
			"none")
				echo ${on:=0} > "${SYSLEDPATH}/brightness"
				;;
			*)
				# Nothing to do
				;;
		esac
	fi
}
 
 
get_logs()
{
	NAME="Logs_${serial}_${shortmac}_$(date "+%Y-%m-%d_%H-%M-%S")"
	tmpdir="/tmp/${NAME}"
 
	mkdir -p ${tmpdir}/
 
	# get logs
	mkdir -p ${tmpdir}/var/log
	cp -a /var/log/* ${tmpdir}/var/log/
 
	# get tmp files 
	mkdir -p ${tmpdir}/tmp
	cp -a /tmp/sys_startup_status.json ${tmpdir}/tmp/
	if [ ${platform} == "IBTS" ]; then
		cp -a /tmp/calib_loraloc.json ${tmpdir}/tmp/
	elif [ ${platform} == "IFEMTO" ]; then
		cp -a /tmp/calib_rf.json ${tmpdir}/tmp/
	fi
 
	# get network config
	mkdir -p ${tmpdir}/etc/network
	mkdir -p ${tmpdir}/etc/firewall.d
	cp -a /etc/network/* ${tmpdir}/etc/network/
	cp -a /etc/firewall.d/* ${tmpdir}/etc/firewall.d/
 
	# CPF config
	mkdir -p ${tmpdir}/etc/default
	mkdir -p ${tmpdir}/user/etc/lorad
	mkdir -p ${tmpdir}/user/etc/lorafwd
	cp -a /etc/default/lorad ${tmpdir}/etc/default/
	cp -a /etc/default/lorafwd ${tmpdir}/etc/default/
	cp -a /user/etc/lorad/* ${tmpdir}/user/etc/lorad/
	cp -a /user/etc/lorafwd/* ${tmpdir}/user/etc/lorafwd/
 
	# miscaellenous config
	mkdir -p ${tmpdir}/etc/rcU.d
	cp -a /etc/rcU.d/* ${tmpdir}/etc/rcU.d
	cp -a /user/.update ${tmpdir}/user/.update 
 
	(
		echo -e "\n\n\n**************  date  *****************"
		date
		echo -e "\n\n\n**************  uptime  *****************"
		uptime
		echo -e "\n\n\n**************  cat /etc/version  *****************"
		cat /etc/version
		echo -e "\n\n\n**************  ps w  *****************"
		ps w
		echo -e "\n\n\n**************  opkg list-installed  *****************"
		opkg list-installed
		echo -e "\n\n\n**************  opkg list  *****************"
		opkg list		
	 	echo -e "\n\n\n**************  monit status  *****************"
	 	monit status
	 	echo -e "\n\n\n**************  mount  *****************"
	 	mount
	) >> ${tmpdir}/cmd_results.txt
 
	(
		echo -e "\n\n\n**************  ifconfig -a  *****************"
		ifconfig -a
		echo -e "\n\n\n**************  gsmdiag.py  *****************"
		gsmdiag.py ; cat /tmp/gsmdiag.txt
		echo -e "\n\n\n**************  route  *****************"
		route
		echo -e "\n\n\n**************  iptables -L  *****************"
		iptables -L
		echo -e "\n\n\n**************  connmanctl technologies  *****************"
		connmanctl technologies
		echo -e "\n\n\n**************  services  *****************"
		connmanctl services
 
		if [ ${platform} == "IBTS" ]; then
			echo -e "\n\n\n**************  qmicli get mode  *****************"
			qmicli -d /dev/cdc-wdm1 --nas-get-system-selection-preference
		fi
	) >> ${tmpdir}/network_result.txt
 
    # get the 25 last NMEA frames; it will take 10s + 25s
	if [ ${platform} == "IBTS" ]; then
 
	      if [ -n "$(lsof | grep $(readlink /dev/nmea2))" ]; then
		      echo " /dev/nmea2 already used by $(lsof | grep $(readlink /dev/nmea2))" > ${tmpdir}/nmea2.log; 
 
	      else
		      timeout -t 10 cat /dev/nmea2 > /dev/null
		      timeout -t 40 head -n 50 /dev/nmea2 > ${tmpdir}/nmea2.log
              fi
	fi
 
	# create an archive with all the files and command results
	# the archive is created on the USB key
	tar -C /tmp -czf ${1}/${NAME}.tar.gz ${NAME}
	echo "${1}/${NAME}.tar.gz"
	sync
 
	rm -rf $tmpdir
}
 
 
#####################
#     MAIN
#####################
 
check_version
 
platform=$(get_platform)
 
reg=$(fslotpreg read GP1)
if [ $? -eq 0 ]; then
	serial=$(printf "%X\n" $reg)
else
	serial="unknown"
fi
shortmac=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}')
 
if [ ${platform} == "IBTS" ]; then
	SYSLEDPATH=${SYSLEDPATH_IBTS}
elif [ ${platform} == "IFEMTO" ]; then
	SYSLEDPATH=${SYSLEDPATH_IFEMTO}
fi
 
# Set led On
set_led none 1
 
# Get logs
get_logs ${USBDIR}
 
# Prepare reboot after USB unplug
cat > /tmp/usbkey_remove << EOF
 
# Switch off Led
echo "none" > ${SYSLEDPATH}/trigger
EOF
 
# All is right, make Led blinking fast to notify User
set_led timer 100 100

Wirnet™ i-series

Sidebar

Table of Contents

Troubleshoot the gateway

Useful logs

Useful commands

Bootcause

Contacting Kerlink's support team

Automatic log gathering

Wirnet™ i-series

User Tools

Site Tools

Sidebar

Table of Contents

Troubleshoot the gateway

Useful logs

Useful commands

Bootcause

Contacting Kerlink's support team

Automatic log gathering

Page Tools