Troubleshoot the gateway

Troubleshoot the gateway

Useful logs

When configuring the gateway or when something unusual happens, most of the time, to understand the behaviour of the gateway it is advised to consult the logs. Here is the list of the most useful logs:

/var/log/messages*: These files are the main logs, they gather the traces of the firmware. They contain the boot traces, the network events, the hardware events, and many other traces.
/var/log/lora.log*: Theses files contain the traces of the common packet forwarder (lorad and lorafwd). If you encounter LoRa issues, this is a good way to start. The verbosity of this log can be increased/decreased. Refer to the log management section.
/tmp/board_info.json: This file contains information about the hardware of the gateway. It is generated at each boot time.
/tmp/calib_loraloc.json (for Wirnet iBTS): This file contains the calibration values specific to this gateway. It is generated at boot time.
/tmp/calib_rf.json (for Wirnet iFemtoCell, Wirnet iFemtoCell-evolution and Wirnet iStation): This file contains the calibration values specific to this gateway. It is generated at boot time.
/dev/nmea* (Wirnet iBTS and Wirnet iStation): These devices receive NMEA frames (GPS) every seconds. It can be monitored with the cat command.

Useful commands

Since Kerlink gateways use a Unix system, most current commands are available (ifconfig, route, ps, iptables, …).

KerOS specific commands:

gsmdiag.py: generates a status of the modem.
connmanctl: used to configure/monitor the network manager.
opkg list-installed: lists the installed packages.
monit status: displays the status of the monitored processes.

Bootcause

The bootcount value is counting the number of reboots since the last power cycle. At each power loss the counter is reset to 0. The maximum value of this counter is 65535.
The bootcount value can be checked in the file /tmp/board_info.json:

# grep bootcount /tmp/board_info.json
        "bootcount": 1,

When a gateway is started/restarted, U-Boot displays the boot cause.
The possible reasons are:

POR: Power On Reset (happens when the gateway is powered or after a powerloss).
WDOG: Watchdog.
SW: Software Reset.
HW: Physical reset (RESET button).
UNKNOWN: unknown cause.

This information is transmitted to initrd and to applications with the kernel argument « bootcause ».
The information is then provided in the bootcause parameter in the file /proc/cmdline.
Example:

# cat /proc/cmdline 
console=ttymxc0,115200 bootmode=nominal bootcause=SW bootcount=5 bootfail=0

Contacting Kerlink's support team

In case you need assistance to troubleshoot your gateway, Kerlink's support team will help you if:

The gateway is still under warranty.
The gateway has been bought directly from Kerlink.

If the gateway does not satisfy this requirement, please contact sales@kerlink.fr. Our Sales team will make you an offer to get assistance.

Otherwise, contact support@kerlink.fr to get assistance. Our support team will open you an account in Kerlink's ticketing tool.

When contacting the support team, to improve and speed up the resolution of your issue, please provide them:

The serial number of your gateway(s):
- For Wirnet iBTS, the serial number is written on a sticker inside the case with “Wirnet iBTS” written on it.
- For Wirnet iFemtoCell, the serial number is written on the rear sticker (it is named “Final product ID”).
- For Wirnet iStation, the serial number is written on the external sticker (it is named “Product ID”).
- For Wirnet iFemtoCell-evolution, the serial number is written on rear sticker (it is named “Product ID”).
A precise description of your problem. “Gateway not working” is not enough to investigate.
The list of the actions you took to troubleshoot/investigate your problem.
The date of your problem.
The version of your firmware.
The logs of your gateway (refer to the automatic log gathering section). Without these logs, it is difficult to investigate. If you are able to reproduce the problem, make sure to reproduce it and gather the logs afterwards.
Any information that would help us understand the context. For example, if the gateway is on the field or in a lab, if a third party program is used, if the gateway was previously working, etc.

Automatic log gathering

To simplify the log gathering process, two scripts are available. These scripts generate an archive containing many logs and commands results.

Gathering logs from WebUI since release 5.9

Gathering logs from shell since release 4.2

Gathering logs from shell on older releases

To gather the logs from shell:

Upload the script bellow (use whatever method you prefer: FileZilla, scp, copy/paste, …) in a folder of your gateway and go to this folder.
Convert the file to Unix text file format using dos2unix get_logs.sh.
Add execution rights to the script using chmod 755 get_logs.sh.
Execute the script using ./get_logs.sh. The execution of the script takes less than 1 minute.
It will generate an archive in the same directory you executed the script. (example: Logs_2E0605F5_7b26_20200124-095206.tar.gz)
Retrieve this archive (use whatever method you prefer: FileZilla, scp, …).

get_logs.sh

#!/bin/sh
 
# Version of the script
SCRIPT_VERSION=1.3
 
# Output directory for the result of the script
OUTPUT_DIR="`dirname $0`"
 
# Execution's environment of the script
SCRIPT_ENV=unknown
 
# PLATFORM: IBTS, IFEMTO, ISTATION, IFEVO
PLATFORM=unknown
# Serial number of the product: 39020003
SERIAL=unknown
# ETH0 MAC: 70:76:ff:01:77:71 => 7771
SHORT_MAC=unknown
 
# LED paths
SYSLEDPATH_4_0_IBTS="/sys/class/leds/yellow:debug"
SYSLEDPATH_4_0_IFEMTO="/sys/class/leds/led1:red:power"
SYSLEDPATH="/sys/class/leds/status"
LED_PATH="/do/not/exist"
 
# Script arguments
LEDS_MANAGEMENT=true
EXEC_COMMANDS=true
 
#
# Print the version of the script
#
show_version() {
	echo "Script version: ${SCRIPT_VERSION}"
}
 
 
#
# Show help
#
show_help()
{
	show_version
	echo -e "Usage: $0 [-h|--help] [-l|--ignore-leds] [-o|--output-dir <directory>]
\t-h|--help: this help
\t-v|--version: version of the script
\t-l|--ignore-leds: this script does not manage the leds (default: autodetection)
\t-o|--output-dir <directory>: force the output <directory> to use for extraction (default: current directory)
\t-c|--disable-commands: disable diagnosis commands (default: executed)
"
}
 
#
# Compare 2 versions to know if inferior or superior
# Res: 0: $1 = $2
#      1: $1 > $2
#      2: $1 < $2
#
compare_versions () {
	if [[ $1 == $2 ]]
	then
		return 0
	fi
	local IFS=.
	local i ver1=($1) ver2=($2)
	# fill empty fields in ver1 with zeros
	for ((i=${#ver1[@]}; i<${#ver2[@]}; i++))
	do
		ver1[i]=0
	done
	for ((i=0; i<${#ver1[@]}; i++))
	do
		if [[ -z ${ver2[i]} ]]
		then
			# fill empty fields in ver2 with zeros
			ver2[i]=0
		fi
		if ((10#${ver1[i]} > 10#${ver2[i]}))
		then
			return 1
		fi
		if ((10#${ver1[i]} < 10#${ver2[i]}))
		then
			return 2
		fi
	done
	return 0
}
 
 
#
# Copy a file or directory if exists
# Arguments:
#            $1: file/directory to copy
#            $2: destination directory (create if necessary)
#
copy_if_exists(){
	local from="$1"
	local to="$2"
 
	if [ -e "${from}" ] ; then
		[ ! -d "${to}" ] && mkdir -p "${to}"
		cp -a "${from}" "${to}"
	fi
}
 
 
#
# Check KerOS/SPN version and stop if:
#  - KerOS < 4.0
#  - SPN < 2.1
#
check_fw_version(){
	# check if /etc/version exists
	if [ -a /etc/version ]; then
		FW_VERSION=$(cat /etc/version)
 
		# Detect type of firmware: KerOS or SPN
		if [[ "${FW_VERSION}" == *-spn-* ]]; then
			# Check if SPN version is not inferior to 2.1, stop if it is the case
			compare_versions ${FW_VERSION:0:3} "2.1"
			if [ $? -eq 2 ]; then
				echo "Your SPN firmware version ($FW_VERSION) is inferior to v2.1. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
				exit
			fi
		else
			# Check if KerOS version is not inferior to 4.0, stop if it is the case
			compare_versions ${FW_VERSION:0:3} "4.0"
			if [ $? -eq 2 ]; then
				echo "Your KerOS firmware version ($FW_VERSION) is inferior to v4. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
				exit
			fi
		fi
	else
		echo "Could not find /etc/version" >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
		exit
	fi
}
 
 
#
# Check platform we are running on
#
detect_platform() {
	PLATFORM="$(fslotpreg getplatform)"
	if [ $? -ne 0 ];
	then
		PLATFORM="NOT_SUPPORTED"
	fi
	echo "Platform detected: ${PLATFORM}"
}
 
 
#
# Get product serial number information: SERIAL and SHORT_MAC
#
fill_serials() {
	local reg=$(fslotpreg read GP1)
	if [ $? -eq 0 ]; then
		SERIAL=$(printf "%X\n" $reg)
	else
		SERIAL="unknown"
	fi
	SHORT_MAC=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}')
}
 
 
#
# Detect the type of environment where the script is launched.
# Possible environments:
#   - udev
#   - shell
#
detect_env() {
	# Detection is only based on the presence or not of the variable DEVNAME
	if [ ! -z ${DEVNAME} ] ; then
		SCRIPT_ENV=udev
		exec > /dev/console 2>&1 # Output the script execution in debug console
	else
		SCRIPT_ENV=shell
		LEDS_MANAGEMENT=false
		OUTPUT_DIR=$(pwd)
	fi
	echo "Script environment: ${SCRIPT_ENV}"
}
 
 
#
# Detect type of leds depending of platform/firmware version
#
detect_leds() {
	if [ -e "${SYSLEDPATH}" ]; then
		LED_PATH="${SYSLEDPATH}"
	elif [ -e "${SYSLEDPATH_4_0_IBTS}" ]; then
		LED_PATH="${SYSLEDPATH_4_0_IBTS}"
	elif [ -e "${SYSLEDPATH_4_0_IFEMTO}" ]; then
		LED_PATH="${SYSLEDPATH_4_0_IFEMTO}"
	fi
}
 
 
#
# Set default led behavior
# $1: trigger
#
#    if trigger = timer : $2 = delay_on, $3 = delay_off
#
set_led() {
	trigger="${1}"
	on="$2"
	off="$3"
 
	if [ -e "${LED_PATH}" ]; then
		echo $trigger > "${LED_PATH}/trigger"
		case "$trigger" in
			"timer")
				echo ${on:=500} > "${LED_PATH}/delay_on"
				echo ${off:=500} > "${LED_PATH}/delay_off"
				;;
			"none")
				echo ${on:=0} > "${LED_PATH}/brightness"
				;;
			*)
				# Nothing to do
				;;
		esac
	fi
}
 
get_nmea_frames() {
	gps_tty="$1"
 
	if ! [ -e "$gps_tty" ]; then
		echo "$gps_tty doesn't exist"
		return 1
	fi
 
	if [ -n "$(lsof | grep $(readlink $gps_tty))" ]; then
		echo "$gps_tty already used by $(lsof | grep $(readlink $gps_tty))"
		return 2
	else
		# Pseudo-term keep only FIRST frames so we keep last of these old frames and get few new frames
		# It will take 3 seconds
		timeout -t 3 cat $gps_tty > /tmp/nmea.log
		cat /tmp/nmea.log
	fi
}
 
#
# Execute some diagnosis commands (can take a long time)
#
execute_commands() {
	echo "Executing diagnosis commands (can take a long time)"
 
	echo -e "\tSystems commands"
		(
		echo -e "\n\n\n**************  fslotpreg getplatform  *****************"
		fslotpreg getplatform
		echo -e "\n\n\n**************  fslotpreg fslotpreg read GP1  *****************"
		fslotpreg read GP1
		echo -e "\n\n\n**************  date  *****************"
		date
		echo -e "\n\n\n**************  uptime  *****************"
		uptime
		echo -e "\n\n\n**************  ps ww *****************"
		ps ww
		echo -e "\n\n\n**************  opkg status  *****************"
		opkg status
		echo -e "\n\n\n**************  monit status  *****************"
		monit status
		echo -e "\n\n\n**************  mount  *****************"
		mount
	) >> ${tmpdir}/cmd_results.txt
 
	echo -e "\tNetwork commands"
	(
		echo -e "\n\n\n**************  ifconfig -a  *****************"
		ifconfig -a
		echo -e "\n\n\n**************  gsmdiag.py  *****************"
		gsmdiag.py ; cat /tmp/gsmdiag.txt
		echo -e "\n\n\n**************  route  *****************"
		route -n
		echo -e "\n\n\n**************  iptables -L  *****************"
		iptables -L
		echo -e "\n\n\n**************  connmanctl technologies  *****************"
		connmanctl technologies
		echo -e "\n\n\n**************  services  *****************"
		connmanctl services
 
		if [ -e /dev/cdc-wdm1 ]; then
			echo -e "\n\n\n**************  qmicli get mode  *****************"
			qmicli -d /dev/cdc-wdm1 --nas-get-system-selection-preference
		fi
	) >> ${tmpdir}/network_result.txt
 
	get_nmea_frames /dev/nmea3 > ${tmpdir}/nmea.log
	[ $? -ne 0 ] && get_nmea_frames /dev/nmea2 >> ${tmpdir}/nmea.log
 
	# Overlay modifications
	echo -e "\tOverlay commands"
	overlay diff -l /.rootfs.ro -u /user/.rootfs_upper/ >> ${tmpdir}/overlay_status.txt << EOF
yes
EOF
 
	echo "End of diagnosis commands"
}
 
 
#
# Get all the logs, configuration files and commands and save it
# in the resquested output directory.
# Arguments:
#            $1: output directory
#
get_logs()
{
	local output_dir="$1"
	local name="Logs_${SERIAL}_${SHORT_MAC}_$(date "+%Y%m%d-%H%M%S")"
	local tmpdir="/user/${name}"
 
	# tmp files
	echo "Extracting tmp files"
	copy_if_exists /tmp/sys_startup_status.json ${tmpdir}/tmp/
	copy_if_exists /tmp/board_info.json ${tmpdir}/tmp/
	copy_if_exists /tmp/calib_loraloc.json ${tmpdir}/tmp/
	copy_if_exists /tmp/calib_rf.json ${tmpdir}/tmp/
 
	# network configuration
	echo "Extracting network configuration"
	copy_if_exists /etc/network/ ${tmpdir}/etc/
	copy_if_exists /etc/firewall.d/ ${tmpdir}/etc/
 
	# CPF (lorad + lorafwd) configuration
	echo "Extracting CPF configuration (if any)"
	for lorad_dir in /user/etc/lorad /etc/lorad
	do
		if [ -d ${lorad_dir} ] ; then
			echo -e "\tExtracting lorad configuration"
			copy_if_exists /etc/default/lorad ${tmpdir}/etc/default/
			copy_if_exists ${lorad_dir} ${tmpdir}${lorad_dir%/*}
		fi
	done
	for lorafwd_dir in /user/etc/lorafwd /etc/lorafwd
	do
		if [ -d ${lorafwd_dir} ] ; then
			echo -e "\tExtracting lorafwd configuration"
			copy_if_exists /etc/default/lorafwd ${tmpdir}/etc/default/
			copy_if_exists ${lorafwd_dir} ${tmpdir}${lorafwd_dir%/*}
		fi
	done
 
	# WMC configuration and logs
	echo "Extracting WMC configuration and logs (if any)"
	if [ -d /user/bscc ] ; then
		echo -e "\tExtracting BSCC configuration and logs"
		copy_if_exists /user/bscc/var.xml ${tmpdir}/user/bscc/
		copy_if_exists /user/bscc/traces/ ${tmpdir}/user/bscc/
	fi
	if [ -d /user/snmp ] ; then
		echo -e "\tExtracting SNMP configuration and logs"
		copy_if_exists /user/snmp/snmpd.conf ${tmpdir}/user/snmp/
		copy_if_exists /user/snmp/traces/ ${tmpdir}/user/snmp/
	fi
 
	# miscellaneous configuration and logs
	echo "Extracting miscellaneous configuration and logs"
	copy_if_exists /etc/rcU.d/ ${tmpdir}/etc/
	copy_if_exists /.update/update.log ${tmpdir}/
	copy_if_exists /user/.update ${tmpdir}/user/
	copy_if_exists /user/.upgrade.log ${tmpdir}/user/
	copy_if_exists /user/.sysupgrade.log ${tmpdir}/user/
	copy_if_exists /etc/version ${tmpdir}/etc/
	copy_if_exists /etc/sysupgrade.conf ${tmpdir}/etc/
	copy_if_exists /etc/sysupgrade.d/ ${tmpdir}/etc/
	copy_if_exists /etc/.squash_layer_version ${tmpdir}/etc/
	echo ${SCRIPT_VERSION} > ${tmpdir}/script_version
 
	if ${EXEC_COMMANDS} ; then
		execute_commands
	fi
 
	# Finish by system logs to prevent space errors
	echo "Extracting system logs"
	copy_if_exists /var/log/ ${tmpdir}/var/
 
	echo "Saving all data in ${output_dir}/${name}.tar.gz"
	tar -C /user -czf ${output_dir}/${name}.tar.gz ${name}
	echo "Done"
	sync
 
	rm -rf $tmpdir
}
 
 
#####################
#     MAIN
#####################
 
# Detect the environment of execution of the script
detect_env
 
# Manage scripts arguments
while [[ $# -gt 0 ]]
do
key="$1"
 
	case $key in
		-l|--ignore-leds)
		LEDS_MANAGEMENT=false
		shift
		;;
		-o|--output-dir)
		OUTPUT_DIR="$2"
		shift 2
		;;
		-c|--disable-commands)
		EXEC_COMMANDS=false
		shift
		;;
		-h|--help)
		show_help
		shift
		exit 1
		;;
		-v|--version)
		show_version
		shift
		exit 1
		;;
		*)
		shift
		;;
	esac
done
 
# Check if firmware version is compatible
check_fw_version
 
# Detect type of platform
detect_platform
 
# Fill SERIAL and SHORT_MAC variables
fill_serials
 
# Set led On
if ${LEDS_MANAGEMENT} ; then
	detect_leds
	set_led none 1
fi
 
# Get logs
get_logs ${OUTPUT_DIR}
 
# End of script, manage user's notifications
if ${LEDS_MANAGEMENT} ; then
	# Prepare reboot after USB unplug
	cat > /tmp/usbkey_remove << EOF
# Switch off Led
echo "none" > ${LED_PATH}/trigger
EOF
 
	# All is right, make Led blinking fast to notify User
	set_led timer 100 100
fi

Gathering logs from USB (no commands required)

To gather logs from a USB without typing any command:

Download the following script and copy it on a USB flash drive.
Create a usbkey.txt file on the USB flash drive as described here.
Plug the USB flash drive into the USB port of the gateway.
Wait for the yellow debug led fast blinking. The execution of the script takes about 1-2 minutes.
Unplug the USB flash drive.
An archive with all the required logs has been created on the USB flash drive.

usb.autorun

#!/bin/sh
 
# Version of the script
SCRIPT_VERSION=1.4
 
# Output directory for the result of the script
OUTPUT_DIR="`dirname $0`"
 
# Execution's environment of the script
SCRIPT_ENV=unknown
 
# PLATFORM: IBTS, IFEMTO, ISTATION, IFEVO
PLATFORM=unknown
# Serial number of the product: 39020003
SERIAL=unknown
# ETH0 MAC: 70:76:ff:01:77:71 => 7771
SHORT_MAC=unknown
 
# LED paths
SYSLEDPATH_4_0_IBTS="/sys/class/leds/yellow:debug"
SYSLEDPATH_4_0_IFEMTO="/sys/class/leds/led1:red:power"
SYSLEDPATH="/sys/class/leds/status"
LED_PATH="/do/not/exist"
 
# Script arguments
LEDS_MANAGEMENT=true
EXEC_COMMANDS=true
 
#
# Print the version of the script
#
show_version() {
        echo "Script version: ${SCRIPT_VERSION}"
}
 
 
#
# Show help
#
show_help()
{
        show_version
        echo -e "Usage: $0 [-h|--help] [-l|--ignore-leds] [-o|--output-dir <directory>]
\t-h|--help: this help
\t-v|--version: version of the script
\t-l|--ignore-leds: this script does not manage the leds (default: autodetection)
\t-o|--output-dir <directory>: force the output <directory> to use for extraction (default: current directory)
\t-c|--disable-commands: disable diagnosis commands (default: executed)
\t-m|--minimal: get minimal logs (to lower generated file size)
"
}
 
#
# Compare 2 versions to know if inferior or superior
# Res: 0: $1 = $2
#      1: $1 > $2
#      2: $1 < $2
#
compare_versions () {
        if [[ $1 == $2 ]]
        then
                return 0
        fi
        local IFS=.
        local i ver1=($1) ver2=($2)
        # fill empty fields in ver1 with zeros
        for ((i=${#ver1[@]}; i<${#ver2[@]}; i++)); do
                ver1[i]=0
        done
        for ((i=0; i<${#ver1[@]}; i++)); do
                if [[ -z ${ver2[i]} ]]
                then
                        # fill empty fields in ver2 with zeros
                        ver2[i]=0
                fi
                if ((10#${ver1[i]} > 10#${ver2[i]}))
                then
                        return 1
                fi
                if ((10#${ver1[i]} < 10#${ver2[i]}))
                then
                        return 2
                fi
        done
        return 0
}
 
 
#
# Copy a file or directory if exists
# Arguments:
#            $1: file/directory to copy
#            $2: destination directory (create if necessary)
#
copy_if_exists(){
        local from="$1"
        local to="$2"
 
        if [ -e "${from}" ] ; then
                [ ! -d "${to}" ] && mkdir -p "${to}"
                cp -a "${from}" "${to}"
        fi
}
 
 
#
# Check KerOS/SPN version and stop if:
#  - KerOS < 4.0
#  - SPN < 2.1
#
check_fw_version(){
        # check if /etc/version exists
        if [ -a /etc/version ]; then
                FW_VERSION=$(cat /etc/version)
 
                # Detect type of firmware: KerOS or SPN
                if [[ "${FW_VERSION}" == *-spn-* ]]; then
                        # Check if SPN version is not inferior to 2.1, stop if it is the case
                        compare_versions ${FW_VERSION:0:3} "2.1"
                        if [ $? -eq 2 ]; then
                                echo "Your SPN firmware version ($FW_VERSION) is inferior to v2.1. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
                                exit
                        fi
                else
                        # Check if KerOS version is not inferior to 4.0, stop if it is the case
                        compare_versions ${FW_VERSION:0:3} "4.0"
                        if [ $? -eq 2 ]; then
                                echo "Your KerOS firmware version ($FW_VERSION) is inferior to v4. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
                                exit
                        fi
                fi
        else
                echo "Could not find /etc/version" >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
                exit
        fi
}
 
 
#
# Check platform we are running on
#
detect_platform() {
        PLATFORM="$(fslotpreg getplatform)"
        if [ $? -ne 0 ];
        then
                PLATFORM="NOT_SUPPORTED"
        fi
        echo "Platform detected: ${PLATFORM}"
}
 
 
#
# Get product serial number information: SERIAL and SHORT_MAC
#
fill_serials() {
        local reg=$(fslotpreg read GP1)
        if [ $? -eq 0 ]; then
                SERIAL=$(printf "%X\n" $reg)
        else
                SERIAL="unknown"
        fi
        if [ -f "/sys/class/net/eth0/address" ]; then
                SHORT_MAC=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}')
        elif [ -f "/sys/class/net/usb0/address" ]; then
                SHORT_MAC=$(cat /sys/class/net/usb0/address | awk -F: '{print $5$6}')
        else
                SHORT_MAC="XXXX"
        fi
}
 
 
#
# Detect the type of environment where the script is launched.
# Possible environments:
#   - udev
#   - shell
#
detect_env() {
        # Detection is only based on the presence or not of the variable DEVNAME
        if [ ! -z ${DEVNAME} ] ; then
                SCRIPT_ENV=udev
                exec > /dev/console 2>&1 # Output the script execution in debug console
        else
                SCRIPT_ENV=shell
                LEDS_MANAGEMENT=false
                OUTPUT_DIR=$(pwd)
        fi
        echo "Script environment: ${SCRIPT_ENV}"
}
 
 
#
# Detect type of leds depending of platform/firmware version
#
detect_leds() {
        if [ -e "${SYSLEDPATH}" ]; then
                LED_PATH="${SYSLEDPATH}"
        elif [ -e "${SYSLEDPATH_4_0_IBTS}" ]; then
                LED_PATH="${SYSLEDPATH_4_0_IBTS}"
        elif [ -e "${SYSLEDPATH_4_0_IFEMTO}" ]; then
                LED_PATH="${SYSLEDPATH_4_0_IFEMTO}"
        fi
}
 
 
#
# Set default led behavior
# $1: trigger
#
#    if trigger = timer : $2 = delay_on, $3 = delay_off
#
set_led() {
        trigger="${1}"
        on="$2"
        off="$3"
 
        if [ -e "${LED_PATH}" ]; then
                echo $trigger > "${LED_PATH}/trigger"
                case "$trigger" in
                        "timer")
                                echo ${on:=500} > "${LED_PATH}/delay_on"
                                echo ${off:=500} > "${LED_PATH}/delay_off"
                                ;;
                        "none")
                                echo ${on:=0} > "${LED_PATH}/brightness"
                                ;;
                        *)
                                # Nothing to do
                                ;;
                esac
        fi
}
 
get_nmea_frames() {
        gps_tty="$1"
 
        if ! [ -e "$gps_tty" ]; then
                echo "$gps_tty doesn't exist"
                return 1
        fi
 
        if [ -n "$(lsof | grep $(readlink $gps_tty))" ]; then
                echo "$gps_tty already used by $(lsof | grep $(readlink $gps_tty))"
                return 2
        else
                # Pseudo-term keep only FIRST frames so we keep last of these old frames and get few new frames
                # It will take 3 seconds
                timeout -t 3 cat $gps_tty > /tmp/nmea.log
                cat /tmp/nmea.log
        fi
}
 
#
# Execute some diagnosis commands (can take a long time)
#
execute_commands() {
        echo "Executing diagnosis commands (can take a long time)"
 
        echo -e "\tSystems commands"
                (
                echo -e "\n\n\n**************  fslotpreg getplatform  *****************"
                fslotpreg getplatform
                echo -e "\n\n\n**************  fslotpreg fslotpreg read GP1  *****************"
                fslotpreg read GP1
                echo -e "\n\n\n**************  date  *****************"
                date
                echo -e "\n\n\n**************  uptime  *****************"
                uptime
                echo -e "\n\n\n**************  ps ww *****************"
                ps ww
                echo -e "\n\n\n**************  opkg status  *****************"
                opkg status
                echo -e "\n\n\n**************  monit status  *****************"
                monit status
                echo -e "\n\n\n**************  mount  *****************"
                mount
                echo -e "\n\n\n**************  OpenVPN Certificate expiration date  *****************"
                compare_versions ${FW_VERSION:0:3} "5.0"
                ret=$?
                if [ $ret -eq 1 ]; then
                        if [ -f "/etc/openvpn/bscc.p12" ]; then
                                openssl pkcs12 -info -in /etc/openvpn/bscc.p12 -nokeys -password file:/etc/openvpn/bscc.password| openssl x509 -noout -enddate
                        fi
                elif [ $ret -eq 2 ]; then
                        grep -iA3 "Validity" /.update/update.log | tail -3 | grep -i "not after" | sed 's/^[ \t]*//'
                fi
                echo -e "\n\n\n**************  Disk info (df -h)  *****************"
                df -h
                echo -e "\n\n\n**************  Log Size (du -sh /var/log)  *****************"
                du -sh /var/log
        ) >> ${tmpdir}/cmd_results.txt
 
        echo -e "\tNetwork commands"
        (
                echo -e "\n\n\n**************  ifconfig -a  *****************"
                ifconfig -a
                echo -e "\n\n\n**************  gsmdiag.py  *****************"
                gsmdiag.py ; cat /tmp/gsmdiag.txt
                echo -e "\n\n\n**************  route  *****************"
                route -n
                echo -e "\n\n\n**************  iptables -L  *****************"
                iptables -L
                echo -e "\n\n\n**************  connmanctl technologies  *****************"
                connmanctl technologies
                echo -e "\n\n\n**************  services  *****************"
                connmanctl services
 
                if [ -e /dev/cdc-wdm1 ]; then
                        echo -e "\n\n\n**************  qmicli get mode  *****************"
                        qmicli -d /dev/cdc-wdm1 --nas-get-system-selection-preference
                fi
        ) >> ${tmpdir}/network_result.txt
 
        get_nmea_frames /dev/nmea3 > ${tmpdir}/nmea.log
        [ $? -ne 0 ] && get_nmea_frames /dev/nmea2 >> ${tmpdir}/nmea.log
 
        # Overlay modifications
        echo -e "\tOverlay commands disabled (KEROS-1907)"
#       overlay diff -l /.rootfs.ro -u /user/.rootfs_upper/ >> ${tmpdir}/overlay_status.txt << EOF
#yes
#EOF
 
        echo "End of diagnosis commands"
}
 
 
#
# Get all the logs, configuration files and commands and save it
# in the resquested output directory.
# Arguments:
#            $1: output directory
#
get_logs()
{
        local output_dir="$1"
        local option="$2"
        local name="Logs_${SERIAL}_${SHORT_MAC}_$(date "+%Y%m%d-%H%M%S")"
        if [ "${option}" = "minimal" ]; then
                name="${name}_minimal"
        fi
        local tmpdir="/user/${name}"
 
        # tmp files
        echo "Extracting tmp files"
        copy_if_exists /tmp/sys_startup_status.json ${tmpdir}/tmp/
        copy_if_exists /tmp/board_info.json ${tmpdir}/tmp/
        copy_if_exists /tmp/calib_loraloc.json ${tmpdir}/tmp/
        copy_if_exists /tmp/calib_rf.json ${tmpdir}/tmp/
 
        # network configuration
        echo "Extracting network configuration"
        if [ "${option}" = "minimal" ]; then
                copy_if_exists /etc/network/connman/ ${tmpdir}/etc/network/
                copy_if_exists /etc/network/ofono/ ${tmpdir}/etc/network/
                copy_if_exists /etc/network/openvpn/ ${tmpdir}/etc/network/
                cp /etc/network/*.conf ${tmpdir}/etc/network/
        else
                copy_if_exists /etc/network/ ${tmpdir}/etc/
        fi
        copy_if_exists /etc/firewall.d/ ${tmpdir}/etc/
 
        # init configuration
        if [ "${option}" != "minimal" ]; then
                echo "Extracting init.d and rc*.d files"
                copy_if_exists /etc/init.d/ ${tmpdir}/etc/
                for filename in /etc/rc*.d; do
                        copy_if_exists ${filename} ${tmpdir}/etc/
                done
        fi
 
        # CPF (lorad + lorafwd) configuration
        echo "Extracting CPF configuration (if any)"
        if [ "${option}" = "minimal" ]; then
                copy_if_exists /user/etc/lorad/lorad.json ${tmpdir}/user/etc/lorad/
                copy_if_exists /etc/lorad/lorad.json ${tmpdir}/etc/lorad/
        else
                for lorad_dir in /user/etc/lorad /etc/lorad; do
                        if [ -d ${lorad_dir} ] ; then
                                echo -e "\tExtracting lorad configuration"
                                copy_if_exists ${lorad_dir} ${tmpdir}${lorad_dir%/*}
                        fi
                done
        fi
        copy_if_exists /etc/default/lorad ${tmpdir}/etc/default/
 
        echo -e "\tExtracting lorafwd configuration"
        if [ "${option}" = "minimal" ]; then
                for filename in /etc/lorafwd/*.toml; do
                        copy_if_exists ${filename} ${tmpdir}/etc/lorafwd/
                done
        else
                for lorafwd_dir in /user/etc/lorafwd /etc/lorafwd; do
                        if [ -d ${lorafwd_dir} ] ; then
                                copy_if_exists ${lorafwd_dir} ${tmpdir}${lorafwd_dir%/*}
                        fi
                done
        fi
        copy_if_exists /etc/lorafwd.toml ${tmpdir}/etc/
        copy_if_exists /etc/default/lorafwd ${tmpdir}/etc/default/
 
        # WMC configuration and logs
        echo "Extracting WMC configuration and logs (if any)"
        if [ -d /user/bscc ] ; then
                echo -e "\tExtracting BSCC configuration and logs"
                copy_if_exists /user/bscc/var.xml ${tmpdir}/user/bscc/
                copy_if_exists /user/bscc/traces/ ${tmpdir}/user/bscc/
        fi
        if [ -d /user/snmp ] ; then
                echo -e "\tExtracting SNMP configuration and logs"
                copy_if_exists /user/snmp/snmpd.conf ${tmpdir}/user/snmp/
                copy_if_exists /user/snmp/traces/ ${tmpdir}/user/snmp/
        fi
        copy_if_exists /etc/default/bscc ${tmpdir}/etc/default/
        copy_if_exists /etc/default/lora-snmp ${tmpdir}/etc/default/
        copy_if_exists /etc/snmp/snmpd.conf ${tmpdir}/etc/snmp/
 
        # basic station configuration and logs
        echo "Extracting basic station configuration and logs (if any)"
        if [ -d /user/basic_station ] ; then
                echo -e "\tExtracting Basic Station configuration and logs"
                copy_if_exists /user/basic_station/etc/ ${tmpdir}/user/basic_station/
                copy_if_exists /user/basic_station/log/ ${tmpdir}/user/basic_station/
        fi
        copy_if_exists /etc/station/ ${tmpdir}/etc/
 
        # miscellaneous configuration and logs
        echo "Extracting miscellaneous configuration and logs"
        copy_if_exists /etc/default/ ${tmpdir}/etc/
        copy_if_exists /etc/openvpn/ ${tmpdir}/etc/
        copy_if_exists /.update/update.log ${tmpdir}/
        copy_if_exists /user/initrd/initrd.log ${tmpdir}/user/initrd
        copy_if_exists /user/.update ${tmpdir}/user/
        copy_if_exists /user/.upgrade.log ${tmpdir}/user/
        copy_if_exists /user/.sysupgrade.log ${tmpdir}/user/
        copy_if_exists /etc/version ${tmpdir}/etc/
        copy_if_exists /etc/sysupgrade.conf ${tmpdir}/etc/
        copy_if_exists /etc/sysupgrade.d/ ${tmpdir}/etc/
        copy_if_exists /etc/.squash_layer_version ${tmpdir}/etc/
        echo ${SCRIPT_VERSION} > ${tmpdir}/script_version
 
        if ${EXEC_COMMANDS} ; then
                execute_commands
        fi
 
        # Finish by system logs to prevent space errors
        echo "Extracting system logs"
        if [ "${option}" = "minimal" ]; then
                copy_if_exists /var/log/messages ${tmpdir}/var/log/
                copy_if_exists /var/log/dmesg ${tmpdir}/var/log/
                for filename in /var/log/*.1; do
                        copy_if_exists ${filename} ${tmpdir}/var/log/
                done
                for filename in /var/log/*.log; do
                        copy_if_exists ${filename} ${tmpdir}/var/log/
                done
        else
                copy_if_exists /var/log/ ${tmpdir}/var/
        fi
 
        echo "Saving all data in ${output_dir}/${name}.tar.gz"
        tar -C /user -czf ${output_dir}/${name}.tar.gz ${name}
        echo "Done"
        sync
 
        rm -rf $tmpdir
}
 
 
#####################
#     MAIN
#####################
 
# Detect the environment of execution of the script
detect_env
 
# Manage scripts arguments
while [[ $# -gt 0 ]]
do
key="$1"
 
        case $key in
                -l|--ignore-leds)
                LEDS_MANAGEMENT=false
                shift
                ;;
                -o|--output-dir)
                OUTPUT_DIR="$2"
                shift 2
                ;;
                -c|--disable-commands)
                EXEC_COMMANDS=false
                shift
                ;;
                -h|--help)
                show_help
                shift
                exit 1
                ;;
                -v|--version)
                show_version
                shift
                exit 1
                ;;
                -m|--minimal)
                OPTION="minimal"
                shift
                ;;
                *)
                shift
                ;;
        esac
done
 
# Check if firmware version is compatible
check_fw_version
 
# Detect type of platform
detect_platform
 
# Fill SERIAL and SHORT_MAC variables
fill_serials
 
# Set led On
if ${LEDS_MANAGEMENT} ; then
        detect_leds
        set_led none 1
fi
 
# Get logs
get_logs ${OUTPUT_DIR} ${OPTION}
 
# End of script, manage user's notifications
if ${LEDS_MANAGEMENT} ; then
        # Prepare reboot after USB unplug
        cat > /tmp/usbkey_remove << EOF
# Switch off Led
echo "none" > ${LED_PATH}/trigger
EOF
 
        # All is right, make Led blinking fast to notify User
        set_led timer 100 100
fi

Helium

This section contains some tips to do the first level of analysis (support level 1) on Helium gateways.
It will help you to investigate before escalate to Kerlink support (level 2).

Please note that there is an existing open wiki for Helium users: Kerlink Helium Hotspot Wiki

First step is to follow Troubleshooting guide chapter which contains:
- Self diagnostic section
- Useful links section

Second step, if necessary, is to follow the table below:

Source of info	KPI	Value	Action
Onboarding dashboard	Last seen	More than 30m ago	Gateway is offline. Ask customer to check power/internet.
	Last seen	Less than 30m ago	Gateway is online. Check next KPI.
	Block height	= 0	Gateway is downloading a snapshot.
		= 1	Gateway is loading the downloaded snapshot.
		Else	KPI is ok, check next KPI.
	Block age	> 30 mn	Gateway is syncing, check again in one hour. If it persists, you can open a ticket to Kerlink Support.
	Block age	< 30 mn	Gateway is synced. Check next KPI.
Helium explorer	24h/14d earnings	> 0 HNT	Seems OK. Confirm with other Explorer KPIs.
	Last activity date	< 3h	Seems OK. Confirm with other Explorer KPIs.
	Is there a constructed challenge in the past 24h?	Yes	Seems OK. Confirm with other Explorer KPIs.
	Is there a constructed challenge in the past 24h?	No	When gateway is synced, it should challenge. You can open a ticket to Kerlink Support.
	Is there a challenged beaconer in the past 24h?	Yes	Seems OK. Confirm with other Explorer KPIs.
	Is there a challenged beaconer in the past 24h?	No	Gateway might be relayed. If not, you can open a ticket to Kerlink Support.
	Is there a witnessed beacon in the past 8-24h?	Value depends on density around miner	Seems OK. Confirm with other Explorer KPIs.
	Is there a witnessed beacon in the past 8-24h?	No	You can open a ticket to Kerlink Support.
	Is there a “broadcasted beacon” in the past 5d?	Yes	Seems OK. Confirm with other Explorer KPIs.
	Is there a “broadcasted beacon” in the past 5d?	No	This is random but it seems odd. You can open a ticket to Kerlink Support.

Gateways connected to a WMC

For gateways connected to a WMC, you can use the Remote Command: Remote Access > Command

Due its cryptographic key and to ensure Kerlink Helium Hotspot security, no SSH access can be given to those gateways.

You can use commands below:

Command	Awaited result
opkg list
monit status miner		Status = OK
monit status lorafwd_helium		Status = OK
/etc/init.d/miner status		Check Diff

If the result is different than the awaited result, you can open a ticket to Kerlink Support.

Table of Contents