Wirnet™ i-series

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Troubleshoot the gateway ======


===== Useful logs =====

When configuring the gateway or when something unusual happens, most of the time, to understand the behaviour of the gateway it is advised to consult the logs.
Here is the list of the most useful logs:

  * ''/var/log/messages*'': These files are the main logs, they gather the traces of the firmware. They contain the boot traces, the network events, the hardware events, and many other traces.
  * ''/var/log/lora.log*'': Theses files contains the traces of the common packet forwarder (lorad and lorafwd). If you encounter LoRa issues, this is a good way to start. The verbosity of this log can be increased/decreased. Refer to the [[ wiki:cpf#log_managment |log managment]] section.
  * ''/tmp/board_info.json'': This file contains information about the hardware of the gateway. It is generated at each boot time.\\
  * ''/tmp/calib_loraloc.json'' (**for Wirnet iBTS**): This file contains the calibration values specific to this gateway. It is generated at boot time.
  * ''/tmp/calib_rf.json'' (**for Wirnet iFemtoCell**): This file contains the calibration values specific to this gateway. It is generated at boot time.
  * ''/dev/nmea*'' (**Wirnet iBTS only**): These devices receive nmea frames (GPS) every seconds. It can be monitored with the ''cat'' command. 

===== Useful commands =====

Since Kerlink gateways use a Unix system, most current commands are available (ifconfig, route, ps, iptables, ...).

**KerOS specific commands:**
  * ''gsmdiag.py'': generates a status of the modem.
  * ''connmanctl'': used to configure/monitor the network manager.
  * ''opkg list-installed'': lists the installed packages.
  * ''monit status'': displays the status of the monitored processes.

===== Bootcause =====

The ''bootcount'' value is counting the number of reboots since the last power cycle.
At each power loss the counter is reset to 0. The maximum value of this counter is 65535.\\
The ''bootcount'' value can be checked in the file ''/tmp/board_info.json'':
<code bash>
# cat /tmp/board_info.json | grep bootcount
        "bootcount": 1,
</code> 

When a gateway is started/restarted, U-Boot displays the boot cause.\\ 
The possible reasons are:

  * POR: Power On Reset (happens when the gateway is powered or after a powerloss).
  * WDOG: Watchdog.
  * SW: Software Reset.
  * HW: Physical reset (RESET button).
  * UNKNOWN: unknown cause.

This information is transmitted to initrd and to applications with the kernel argument « bootcause ». \\
The information is then provided in the ''bootcause'' parameter in the file **''/proc/cmdline''**. \\
Example:
<code bash>
# cat /proc/cmdline 
console=ttymxc0,115200 bootmode=nominal bootcause=SW bootcount=5 bootfail=0
</code>

===== Contacting Kerlink's support team =====

In case you need assistance to troubleshoot your gateway, Kerlink's support team will help you if:
  * The gateway is still under warranty.
  * The gateway has been bought directly from Kerlink.

If the gateway does not satisfy this requirement, please contact [[sales@kerlink.fr]]. Our Sales team will make you an offer to get assistance.

Otherwise, contact [[support@kerlink.fr]] to get assistance. Our support team will open you an account in Kerlink's ticketing tool.

When contacting the support team, to improve and speed up the resolution of your issue, please provide them:
  * The serial number of your gateway(s) (the serial number is written on a sticker inside the case with "Wirnet iBTS" written on it or the "Final product ID" on the rear sticker for Wirnet iFemtoCell).
  * A precise description of your problem. "Gateway not working" is not enough to investigate.
  * The list of the actions you took to troubleshoot/investigate your problem.
  * The date of your problem.
  * The version of your firmware.
  * The logs of your gateway (refer to the automatic log gathering section). Without these logs, it is difficult to investigate. If you are able to reproduce the problem, make sure to reproduce it and gather the logs afterward.
  * Any information that would help us understand the context. For example, if the gateway is on the field or in a lab, if a third party program is used, if the gateway was previously working, etc.

===== Automatic log gathering =====

To simplify the log gathering process, two scripts are available. These scripts generate an archive containing many logs and commands results.

<note tip>
TODO 
  * adapt for CPF logs
  * add result of overlay diff
  * add mount command
  * remove reference to /user/rootfs_rw
</note>

++++Gathering logs from shell|

To gather the logs from shell:
  * Upload the script bellow (use whatever method you prefer: FileZilla, scp, copy/paste, ...) in a folder of your gateway and go to this folder.
  * Convert the file to Unix text file format using ''dos2unix gather_logs.sh''.
  * Add execution rights to the script using ''chmod 755 gather_logs.sh''.
  * Execute the script using ''./gather_logs.sh''. //The execution of the script takes about 1-2 minutes.//
  * It will generate an archive under ''/tmp''.
  * Retrieve this archive (use whatever method you prefer: FileZilla, scp, ...).

<code bash gather_logs.sh>
#!/bin/sh

# get the name and the MAC of the board
reg=$(cat /sys/fsl_otp/HW_OCOTP_GP1)
serial=$(printf "%X\n" $reg)
shortmac=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}')

NAME="Logs_${serial}_${shortmac}_$(date "+%Y%m%d-%H%M%S")"
tmpdir="/tmp/${NAME}"

# generate a temporary directory
mkdir -p ${tmpdir}/

{
	# get general information
	{ uname -a; echo; date; echo; uptime; } > ${tmpdir}/status.txt
	ps ww > ${tmpdir}/ps.log

	# get /var/log logs
	mkdir -p ${tmpdir}/var/log/
	cp -a /var/log/* ${tmpdir}/var/log/

	# get packet forwarder logs/configuration
	shopt -s extglob
	mkdir -p ${tmpdir}/packet_forwarder/
	cp -a /user/!(rootfs_rw)/var/log/* ${tmpdir}/packet_forwarder/
	cp -a /user/!(rootfs_rw)/etc/* ${tmpdir}/packet_forwarder/		

	# get /tmp files
	cp /tmp/sys_startup_status.json ${tmpdir}/
	cp /tmp/calib_loraloc.json ${tmpdir}/
	cp /tmp/gps_vers.txt ${tmpdir}/

	# get user configuration
	mkdir -p ${tmpdir}/config/
	cp -a /user/rootfs_rw/etc/* ${tmpdir}/config/

	# get network information
	ifconfig -a > ${tmpdir}/ifconfig.log
	route > ${tmpdir}/route.log
	iptables -L > ${tmpdir}/iptables.log
	netstat > ${tmpdir}/netstat.log
	{ echo 'connmanctl technologies'
	  connmanctl technologies

	  echo
	  echo 'connmanctl services'
	  connmanctl services
	} > ${tmpdir}/connmanctl.log

	# get GSM information
	gsmdiag.py ; cp /tmp/gsmdiag.txt ${tmpdir}/
	{ echo 'qmicli -d /dev/cdc-wdm0 --dms-get-ids'
	  qmicli -d /dev/cdc-wdm0 --dms-get-ids

	  echo 	
	  echo 'qmicli -d /dev/cdc-wdm0 --nas-get-serving-system'
	  qmicli -d /dev/cdc-wdm0 --nas-get-serving-system

	  echo 	
	  echo 'qmicli -d /dev/cdc-wdm0 --nas-get-system-selection-preference'
	  qmicli -d /dev/cdc-wdm0 --nas-get-system-selection-preference
	  
	  echo 	
	  echo 'qmicli -d /dev/cdc-wdm0 --dms-get-operating-mode'
	  qmicli -d /dev/cdc-wdm0 --dms-get-operating-mode
	} > ${tmpdir}/qmicli_0.log

	# get GPS info
	timeout -t 10 cat /dev/nmea2 > /dev/null
	head -n 50 /dev/nmea2 > ${tmpdir}/nmea2.log

	# get LoRa information
	loraloc_info > ${tmpdir}/loraloc_info.log

	# get miscellaneous information
	find /user/ > ${tmpdir}/user_file_list.txt
	opkg list-installed > ${tmpdir}/installed-packages.log	
        monit status > ${tmpdir}/monit_status.log	
} > ${tmpdir}/gather_logs.log 2>&1

tar -C /tmp -czf /tmp/${NAME}.tar.gz ${NAME}
echo "/tmp/${NAME}.tar.gz created"
</code>

++++

++++Gathering logs from USB (no commands required)|

To gather logs from a USB without typing any command:
  * Download the following script and copy it on a USB key.
  * Create a ''usbkey.txt'' file on the USB key as described [[wiki:sw_updates#specific_files|here]].
  * Plug the USB key on the CPU board.
  * Wait for the yellow debug led fast blinking. //The execution of the script takes about 1-2 minutes.//
  * Unplug the USB key.
  * An archive with all the required logs has been created on the USB key.

<code bash usb.autorun>
#!/bin/sh

USBDIR="`dirname $0`"

SYSLEDPATH_IBTS="/sys/class/leds/yellow:debug/"
SYSLEDPATH_IFEMTO="/sys/class/leds/led1:red:power/"


#
# Check platform we are running on
#
get_platform() {
	platform="$(fslotpreg getplatform)"
	if [ $? -ne 0 ];
	then
		platform="NOT_SUPPORTED"
	fi
	echo ${platform}
}


#
# Set default led behavior
# $1: trigger
#
#    if trigger = timer : $2 = delay_on, $3 = delay_off
#
set_led() {
	trigger="${1}"
	on="$2"
	off="$3"

	if [ -d "${SYSLEDPATH}" ]; then
		echo $trigger > "${SYSLEDPATH}/trigger"
		case "$trigger" in
			"timer")
				echo ${on:=500} > "${SYSLEDPATH}/delay_on"
				echo ${off:=500} > "${SYSLEDPATH}/delay_off"
				;;
			"none")
				echo ${on:=0} > "${SYSLEDPATH}/brightness"
				;;
			*)
				# Nothing to do
				;;
		esac
	fi
}


get_logs()
{
	NAME="Logs_${serial}_${shortmac}_$(date "+%Y-%m-%d_%H-%M-%S")"
	tmpdir="/tmp/${NAME}"

	mkdir -p ${tmpdir}/

	# get logs
	mkdir -p ${tmpdir}/var/log
	cp -a /var/log/* ${tmpdir}/var/log/

	# get tmp files 
	mkdir -p ${tmpdir}/tmp
	cp -a /tmp/sys_startup_status.json ${tmpdir}/tmp/
	if [ ${platform} == "IBTS" ]; then
		cp -a /tmp/calib_loraloc.json ${tmpdir}/tmp/
	elif [ ${platform} == "IFEMTO" ]; then
		cp -a /tmp/calib_rf.json ${tmpdir}/tmp/
	fi

	# get network config
	mkdir -p ${tmpdir}/etc/network
	mkdir -p ${tmpdir}/etc/firewall.d
	cp -a /etc/network/* ${tmpdir}/etc/network/
	cp -a /etc/firewall.d/* ${tmpdir}/etc/firewall.d/

	# CPF config
	mkdir -p ${tmpdir}/etc/default
	mkdir -p ${tmpdir}/user/etc/lorad
	mkdir -p ${tmpdir}/user/etc/lorafwd
	cp -a /etc/default/lorad ${tmpdir}/etc/default/
	cp -a /etc/default/lorafwd ${tmpdir}/etc/default/
	cp -a /user/etc/lorad/* ${tmpdir}/user/etc/lorad/
	cp -a /user/etc/lorafwd/* ${tmpdir}/user/etc/lorafwd/

	# miscaellenous config
	mkdir -p ${tmpdir}/etc/rcU.d
	cp -a /etc/rcU.d/* ${tmpdir}/etc/rcU.d
	cp -a /user/.update ${tmpdir}/user/.update 

	(
		echo -e "\n\n\n**************  date  *****************"
		date
		echo -e "\n\n\n**************  uptime  *****************"
		uptime
		echo -e "\n\n\n**************  cat /etc/version  *****************"
		cat /etc/version
		echo -e "\n\n\n**************  ps w  *****************"
		ps w
		echo -e "\n\n\n**************  opkg list-installed  *****************"
		opkg list-installed
		echo -e "\n\n\n**************  opkg list  *****************"
		opkg list		
	 	echo -e "\n\n\n**************  monit status  *****************"
	 	monit status
	) >> ${tmpdir}/cmd_results.txt

	(
		echo -e "\n\n\n**************  ifconfig -a  *****************"
		ifconfig -a
		echo -e "\n\n\n**************  gsmdiag.py  *****************"
		gsmdiag.py ; cat /tmp/gsmdiag.txt
		echo -e "\n\n\n**************  route  *****************"
		route
		echo -e "\n\n\n**************  iptables -L  *****************"
		iptables -L
		echo -e "\n\n\n**************  connmanctl technologies  *****************"
		connmanctl technologies
		echo -e "\n\n\n**************  services  *****************"
		connmanctl services

		if [ ${platform} == "IBTS" ]; then
			echo -e "\n\n\n**************  qmicli get mode  *****************"
			qmicli -d /dev/cdc-wdm1 --nas-get-system-selection-preference
		fi
	) >> ${tmpdir}/network_result.txt
	
    # get the 25 last NMEA frames; it will take 10s + 25s
	if [ ${platform} == "IBTS" ]; then
		timeout -t 10 cat /dev/nmea2 > /dev/null
		head -n 50 /dev/nmea2 > ${tmpdir}/nmea2.log
	fi

	# create an archive with all the files and command results
	# the archive is created on the USB key
	tar -C /tmp -czf ${1}/${NAME}.tar.gz ${NAME}
	echo "${1}/${NAME}.tar.gz"
	sync

	rm -rf $tmpdir
}


#####################
#     MAIN
#####################

platform=$(get_platform)

reg=$(fslotpreg read GP1)
if [ $? -eq 0 ]; then
	serial=$(printf "%X\n" $reg)
else
	serial="unknown"
fi
shortmac=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}')

if [ ${platform} == "IBTS" ]; then
	SYSLEDPATH=${SYSLEDPATH_IBTS}
elif [ ${platform} == "IFEMTO" ]; then
	SYSLEDPATH=${SYSLEDPATH_IFEMTO}
fi

# Set led On
set_led none 1

# Get logs
get_logs ${USBDIR}

# Prepare reboot after USB unplug
cat > /tmp/usbkey_remove << EOF

# Switch off Led
echo "none" > ${SYSLEDPATH}/trigger
EOF

# All is right, make Led blinking fast to notify User
set_led timer 100 100
</code>

++++
Wirnet™ i-series

User Tools

Site Tools

Page Tools
