Wirnet™ i-series

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Troubleshoot the gateway ======


===== Useful logs =====

When configuring the gateway or when something unusual happens, most of the time, to understand the behaviour of the gateway it is advised to consult the logs.
Here is the list of the most useful logs:

  * ''/var/log/messages*'': These files are the main logs, they gather the traces of the firmware. They contain the boot traces, the network events, the hardware events, and many other traces.
  * ''/var/log/lora.log*'': Theses files contain the traces of the common packet forwarder (lorad and lorafwd). If you encounter LoRa issues, this is a good way to start. The verbosity of this log can be increased/decreased. Refer to the [[ wiki:lora:cpf#log_management |log management]] section.
  * ''/tmp/board_info.json'': This file contains information about the hardware of the gateway. It is generated at each boot time.\\
  * ''/tmp/calib_loraloc.json'' (**for Wirnet iBTS**): This file contains the calibration values specific to this gateway. It is generated at boot time.
  * ''/tmp/calib_rf.json'' (**for Wirnet iFemtoCell, Wirnet iFemtoCell-evolution and Wirnet iStation**): This file contains the calibration values specific to this gateway. It is generated at boot time.
  * ''/dev/nmea*'' (**Wirnet iBTS and Wirnet iStation**): These devices receive NMEA frames (GPS) every seconds. It can be monitored with the ''cat'' command. 

===== Useful commands =====

Since Kerlink gateways use a Unix system, most current commands are available (''ifconfig'', ''route'', ''ps'', ''iptables'', ...).

**KerOS specific commands:**
  * ''gsmdiag.py'': generates a status of the modem.
  * ''connmanctl'': used to configure/monitor the network manager.
  * ''opkg list-installed'': lists the installed packages.
  * ''monit status'': displays the status of the monitored processes.

===== Bootcause =====

The ''bootcount'' value is counting the number of reboots since the last power cycle.
At each power loss the counter is reset to 0. The maximum value of this counter is 65535.\\
The ''bootcount'' value can be checked in the file ''/tmp/board_info.json'':
<code bash>
# grep bootcount /tmp/board_info.json
        "bootcount": 1,
</code> 

When a gateway is started/restarted, U-Boot displays the boot cause.\\ 
The possible reasons are:

  * POR: Power On Reset (happens when the gateway is powered or after a powerloss).
  * WDOG: Watchdog.
  * SW: Software Reset.
  * HW: Physical reset (RESET button).
  * UNKNOWN: unknown cause.

This information is transmitted to initrd and to applications with the kernel argument « bootcause ». \\
The information is then provided in the ''bootcause'' parameter in the file **''/proc/cmdline''**. \\
Example:
<code bash>
# cat /proc/cmdline 
console=ttymxc0,115200 bootmode=nominal bootcause=SW bootcount=5 bootfail=0
</code>

===== Contacting Kerlink's support team =====

In case you need assistance to troubleshoot your gateway, Kerlink's support team will help you if:
  * The gateway is still under warranty.
  * The gateway has been bought directly from Kerlink.

If the gateway does not satisfy this requirement, please contact [[sales@kerlink.fr]]. Our Sales team will make you an offer to get assistance.

Otherwise, contact [[support@kerlink.fr]] to get assistance. Our support team will open you an account in Kerlink's ticketing tool.

When contacting the support team, to improve and speed up the resolution of your issue, please provide them:
  * The serial number of your gateway(s):
    * For Wirnet iBTS, the serial number is written on a sticker inside the case with "Wirnet iBTS" written on it.
    * For Wirnet iFemtoCell, the serial number is written on the rear sticker (it is named "Final product ID").
    * For Wirnet iStation, the serial number is written on the external sticker (it is named "Product ID"). 
    * For Wirnet iFemtoCell-evolution, the serial number is written on rear sticker (it is named "Product ID"). 
  * A precise description of your problem. "Gateway not working" is not enough to investigate.
  * The list of the actions you took to troubleshoot/investigate your problem.
  * The date of your problem.
  * The version of your firmware.
  * The logs of your gateway (refer to the automatic log gathering section). Without these logs, it is difficult to investigate. If you are able to reproduce the problem, make sure to reproduce it and gather the logs afterwards.
  * Any information that would help us understand the context. For example, if the gateway is on the field or in a lab, if a third party program is used, if the gateway was previously working, etc.

===== Automatic log gathering =====

To simplify the log gathering process, two scripts are available. These scripts generate an archive containing many logs and commands results.

++++Gathering logs from WebUI since release 5.9|
[[wiki:systeme_mana:webui|Section Gateway administration menu]]
++++

++++Gathering logs from shell since release 4.2|

To gather the logs from shell:
  * Execute the command ''get_logs''. //The execution of the script takes less than 1 minute.//
  * It will generate an archive in the same directory you executed the script (example: ''Logs_2E0605F5_7b26_20200124-095206.tar.gz'')
  * Retrieve this archive (use whatever method you prefer: FileZilla, scp, ...).

<note>**To use only in case of speed or data restriction**, there is a new option to gather lighter logs: ''get_logs -m'' or ''get_logs --minimal''. \\
**Please note that it will only gather latest logs** (.log & .log.1 files). \\
It will generate an light archive recognizable by its name: _minimal.tar.gz (example: ''Logs_3903002E_0091_20230419-134556_minimal.tar.gz'')
</note>
++++

++++Gathering logs from shell on older releases|

To gather the logs from shell:
  * Upload the script bellow (use whatever method you prefer: FileZilla, scp, copy/paste, ...) in a folder of your gateway and go to this folder.
  * Convert the file to Unix text file format using ''dos2unix get_logs.sh''.
  * Add execution rights to the script using ''chmod 755 get_logs.sh''.
  * Execute the script using ''./get_logs.sh''. //The execution of the script takes less than 1 minute.//
  * It will generate an archive in the same directory you executed the script. (example: ''Logs_2E0605F5_7b26_20200124-095206.tar.gz'')
  * Retrieve this archive (use whatever method you prefer: FileZilla, scp, ...).


<code bash get_logs.sh>
#!/bin/sh

# Version of the script
SCRIPT_VERSION=1.3

# Output directory for the result of the script
OUTPUT_DIR="`dirname $0`"

# Execution's environment of the script
SCRIPT_ENV=unknown

# PLATFORM: IBTS, IFEMTO, ISTATION, IFEVO
PLATFORM=unknown
# Serial number of the product: 39020003
SERIAL=unknown
# ETH0 MAC: 70:76:ff:01:77:71 => 7771
SHORT_MAC=unknown

# LED paths
SYSLEDPATH_4_0_IBTS="/sys/class/leds/yellow:debug"
SYSLEDPATH_4_0_IFEMTO="/sys/class/leds/led1:red:power"
SYSLEDPATH="/sys/class/leds/status"
LED_PATH="/do/not/exist"

# Script arguments
LEDS_MANAGEMENT=true
EXEC_COMMANDS=true

#
# Print the version of the script
#
show_version() {
	echo "Script version: ${SCRIPT_VERSION}"
}


#
# Show help
#
show_help()
{
	show_version
	echo -e "Usage: $0 [-h|--help] [-l|--ignore-leds] [-o|--output-dir <directory>]
\t-h|--help: this help
\t-v|--version: version of the script
\t-l|--ignore-leds: this script does not manage the leds (default: autodetection)
\t-o|--output-dir <directory>: force the output <directory> to use for extraction (default: current directory)
\t-c|--disable-commands: disable diagnosis commands (default: executed)
"
}

#
# Compare 2 versions to know if inferior or superior
# Res: 0: $1 = $2
#      1: $1 > $2
#      2: $1 < $2
#
compare_versions () {
	if [[ $1 == $2 ]]
	then
		return 0
	fi
	local IFS=.
	local i ver1=($1) ver2=($2)
	# fill empty fields in ver1 with zeros
	for ((i=${#ver1[@]}; i<${#ver2[@]}; i++))
	do
		ver1[i]=0
	done
	for ((i=0; i<${#ver1[@]}; i++))
	do
		if [[ -z ${ver2[i]} ]]
		then
			# fill empty fields in ver2 with zeros
			ver2[i]=0
		fi
		if ((10#${ver1[i]} > 10#${ver2[i]}))
		then
			return 1
		fi
		if ((10#${ver1[i]} < 10#${ver2[i]}))
		then
			return 2
		fi
	done
	return 0
}


#
# Copy a file or directory if exists
# Arguments:
#            $1: file/directory to copy
#            $2: destination directory (create if necessary)
#
copy_if_exists(){
	local from="$1"
	local to="$2"

	if [ -e "${from}" ] ; then
		[ ! -d "${to}" ] && mkdir -p "${to}"
		cp -a "${from}" "${to}"
	fi
}


#
# Check KerOS/SPN version and stop if:
#  - KerOS < 4.0
#  - SPN < 2.1
#
check_fw_version(){
	# check if /etc/version exists
	if [ -a /etc/version ]; then
		FW_VERSION=$(cat /etc/version)

		# Detect type of firmware: KerOS or SPN
		if [[ "${FW_VERSION}" == *-spn-* ]]; then
			# Check if SPN version is not inferior to 2.1, stop if it is the case
			compare_versions ${FW_VERSION:0:3} "2.1"
			if [ $? -eq 2 ]; then
				echo "Your SPN firmware version ($FW_VERSION) is inferior to v2.1. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
				exit
			fi
		else
			# Check if KerOS version is not inferior to 4.0, stop if it is the case
			compare_versions ${FW_VERSION:0:3} "4.0"
			if [ $? -eq 2 ]; then
				echo "Your KerOS firmware version ($FW_VERSION) is inferior to v4. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
				exit
			fi
		fi
	else
		echo "Could not find /etc/version" >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
		exit
	fi
}


#
# Check platform we are running on
#
detect_platform() {
	PLATFORM="$(fslotpreg getplatform)"
	if [ $? -ne 0 ];
	then
		PLATFORM="NOT_SUPPORTED"
	fi
	echo "Platform detected: ${PLATFORM}"
}


#
# Get product serial number information: SERIAL and SHORT_MAC
#
fill_serials() {
	local reg=$(fslotpreg read GP1)
	if [ $? -eq 0 ]; then
		SERIAL=$(printf "%X\n" $reg)
	else
		SERIAL="unknown"
	fi
	SHORT_MAC=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}')
}


#
# Detect the type of environment where the script is launched.
# Possible environments:
#   - udev
#   - shell
#
detect_env() {
	# Detection is only based on the presence or not of the variable DEVNAME
	if [ ! -z ${DEVNAME} ] ; then
		SCRIPT_ENV=udev
		exec > /dev/console 2>&1 # Output the script execution in debug console
	else
		SCRIPT_ENV=shell
		LEDS_MANAGEMENT=false
		OUTPUT_DIR=$(pwd)
	fi
	echo "Script environment: ${SCRIPT_ENV}"
}


#
# Detect type of leds depending of platform/firmware version
#
detect_leds() {
	if [ -e "${SYSLEDPATH}" ]; then
		LED_PATH="${SYSLEDPATH}"
	elif [ -e "${SYSLEDPATH_4_0_IBTS}" ]; then
		LED_PATH="${SYSLEDPATH_4_0_IBTS}"
	elif [ -e "${SYSLEDPATH_4_0_IFEMTO}" ]; then
		LED_PATH="${SYSLEDPATH_4_0_IFEMTO}"
	fi
}


#
# Set default led behavior
# $1: trigger
#
#    if trigger = timer : $2 = delay_on, $3 = delay_off
#
set_led() {
	trigger="${1}"
	on="$2"
	off="$3"

	if [ -e "${LED_PATH}" ]; then
		echo $trigger > "${LED_PATH}/trigger"
		case "$trigger" in
			"timer")
				echo ${on:=500} > "${LED_PATH}/delay_on"
				echo ${off:=500} > "${LED_PATH}/delay_off"
				;;
			"none")
				echo ${on:=0} > "${LED_PATH}/brightness"
				;;
			*)
				# Nothing to do
				;;
		esac
	fi
}

get_nmea_frames() {
	gps_tty="$1"

	if ! [ -e "$gps_tty" ]; then
		echo "$gps_tty doesn't exist"
		return 1
	fi

	if [ -n "$(lsof | grep $(readlink $gps_tty))" ]; then
		echo "$gps_tty already used by $(lsof | grep $(readlink $gps_tty))"
		return 2
	else
		# Pseudo-term keep only FIRST frames so we keep last of these old frames and get few new frames
		# It will take 3 seconds
		timeout -t 3 cat $gps_tty > /tmp/nmea.log
		cat /tmp/nmea.log
	fi
}

#
# Execute some diagnosis commands (can take a long time)
#
execute_commands() {
	echo "Executing diagnosis commands (can take a long time)"

	echo -e "\tSystems commands"
		(
		echo -e "\n\n\n**************  fslotpreg getplatform  *****************"
		fslotpreg getplatform
		echo -e "\n\n\n**************  fslotpreg fslotpreg read GP1  *****************"
		fslotpreg read GP1
		echo -e "\n\n\n**************  date  *****************"
		date
		echo -e "\n\n\n**************  uptime  *****************"
		uptime
		echo -e "\n\n\n**************  ps ww *****************"
		ps ww
		echo -e "\n\n\n**************  opkg status  *****************"
		opkg status
		echo -e "\n\n\n**************  monit status  *****************"
		monit status
		echo -e "\n\n\n**************  mount  *****************"
		mount
	) >> ${tmpdir}/cmd_results.txt

	echo -e "\tNetwork commands"
	(
		echo -e "\n\n\n**************  ifconfig -a  *****************"
		ifconfig -a
		echo -e "\n\n\n**************  gsmdiag.py  *****************"
		gsmdiag.py ; cat /tmp/gsmdiag.txt
		echo -e "\n\n\n**************  route  *****************"
		route -n
		echo -e "\n\n\n**************  iptables -L  *****************"
		iptables -L
		echo -e "\n\n\n**************  connmanctl technologies  *****************"
		connmanctl technologies
		echo -e "\n\n\n**************  services  *****************"
		connmanctl services

		if [ -e /dev/cdc-wdm1 ]; then
			echo -e "\n\n\n**************  qmicli get mode  *****************"
			qmicli -d /dev/cdc-wdm1 --nas-get-system-selection-preference
		fi
	) >> ${tmpdir}/network_result.txt

	get_nmea_frames /dev/nmea3 > ${tmpdir}/nmea.log
	[ $? -ne 0 ] && get_nmea_frames /dev/nmea2 >> ${tmpdir}/nmea.log

	# Overlay modifications
	echo -e "\tOverlay commands"
	overlay diff -l /.rootfs.ro -u /user/.rootfs_upper/ >> ${tmpdir}/overlay_status.txt << EOF
yes
EOF

	echo "End of diagnosis commands"
}


#
# Get all the logs, configuration files and commands and save it
# in the resquested output directory.
# Arguments:
#            $1: output directory
#
get_logs()
{
	local output_dir="$1"
	local name="Logs_${SERIAL}_${SHORT_MAC}_$(date "+%Y%m%d-%H%M%S")"
	local tmpdir="/user/${name}"

	# tmp files
	echo "Extracting tmp files"
	copy_if_exists /tmp/sys_startup_status.json ${tmpdir}/tmp/
	copy_if_exists /tmp/board_info.json ${tmpdir}/tmp/
	copy_if_exists /tmp/calib_loraloc.json ${tmpdir}/tmp/
	copy_if_exists /tmp/calib_rf.json ${tmpdir}/tmp/

	# network configuration
	echo "Extracting network configuration"
	copy_if_exists /etc/network/ ${tmpdir}/etc/
	copy_if_exists /etc/firewall.d/ ${tmpdir}/etc/

	# CPF (lorad + lorafwd) configuration
	echo "Extracting CPF configuration (if any)"
	for lorad_dir in /user/etc/lorad /etc/lorad
	do
		if [ -d ${lorad_dir} ] ; then
			echo -e "\tExtracting lorad configuration"
			copy_if_exists /etc/default/lorad ${tmpdir}/etc/default/
			copy_if_exists ${lorad_dir} ${tmpdir}${lorad_dir%/*}
		fi
	done
	for lorafwd_dir in /user/etc/lorafwd /etc/lorafwd
	do
		if [ -d ${lorafwd_dir} ] ; then
			echo -e "\tExtracting lorafwd configuration"
			copy_if_exists /etc/default/lorafwd ${tmpdir}/etc/default/
			copy_if_exists ${lorafwd_dir} ${tmpdir}${lorafwd_dir%/*}
		fi
	done

	# WMC configuration and logs
	echo "Extracting WMC configuration and logs (if any)"
	if [ -d /user/bscc ] ; then
		echo -e "\tExtracting BSCC configuration and logs"
		copy_if_exists /user/bscc/var.xml ${tmpdir}/user/bscc/
		copy_if_exists /user/bscc/traces/ ${tmpdir}/user/bscc/
	fi
	if [ -d /user/snmp ] ; then
		echo -e "\tExtracting SNMP configuration and logs"
		copy_if_exists /user/snmp/snmpd.conf ${tmpdir}/user/snmp/
		copy_if_exists /user/snmp/traces/ ${tmpdir}/user/snmp/
	fi

	# miscellaneous configuration and logs
	echo "Extracting miscellaneous configuration and logs"
	copy_if_exists /etc/rcU.d/ ${tmpdir}/etc/
	copy_if_exists /.update/update.log ${tmpdir}/
	copy_if_exists /user/.update ${tmpdir}/user/
	copy_if_exists /user/.upgrade.log ${tmpdir}/user/
	copy_if_exists /user/.sysupgrade.log ${tmpdir}/user/
	copy_if_exists /etc/version ${tmpdir}/etc/
	copy_if_exists /etc/sysupgrade.conf ${tmpdir}/etc/
	copy_if_exists /etc/sysupgrade.d/ ${tmpdir}/etc/
	copy_if_exists /etc/.squash_layer_version ${tmpdir}/etc/
	echo ${SCRIPT_VERSION} > ${tmpdir}/script_version

	if ${EXEC_COMMANDS} ; then
		execute_commands
	fi

	# Finish by system logs to prevent space errors
	echo "Extracting system logs"
	copy_if_exists /var/log/ ${tmpdir}/var/

	echo "Saving all data in ${output_dir}/${name}.tar.gz"
	tar -C /user -czf ${output_dir}/${name}.tar.gz ${name}
	echo "Done"
	sync

	rm -rf $tmpdir
}


#####################
#     MAIN
#####################

# Detect the environment of execution of the script
detect_env

# Manage scripts arguments
while [[ $# -gt 0 ]]
do
key="$1"

	case $key in
		-l|--ignore-leds)
		LEDS_MANAGEMENT=false
		shift
		;;
		-o|--output-dir)
		OUTPUT_DIR="$2"
		shift 2
		;;
		-c|--disable-commands)
		EXEC_COMMANDS=false
		shift
		;;
		-h|--help)
		show_help
		shift
		exit 1
		;;
		-v|--version)
		show_version
		shift
		exit 1
		;;
		*)
		shift
		;;
	esac
done

# Check if firmware version is compatible
check_fw_version

# Detect type of platform
detect_platform

# Fill SERIAL and SHORT_MAC variables
fill_serials

# Set led On
if ${LEDS_MANAGEMENT} ; then
	detect_leds
	set_led none 1
fi

# Get logs
get_logs ${OUTPUT_DIR}

# End of script, manage user's notifications
if ${LEDS_MANAGEMENT} ; then
	# Prepare reboot after USB unplug
	cat > /tmp/usbkey_remove << EOF
# Switch off Led
echo "none" > ${LED_PATH}/trigger
EOF

	# All is right, make Led blinking fast to notify User
	set_led timer 100 100
fi
</code>



++++

++++Gathering logs from USB (no commands required)|

To gather logs from a USB without typing any command:
  * Download the following script and copy it on a USB flash drive.
  * Create a ''usbkey.txt'' file on the USB flash drive as described [[wiki:keros_custo:sw_updates#specific_files|here]].
  * Plug the USB flash drive into the USB port of the gateway.
  * Wait for the yellow debug led fast blinking. //The execution of the script takes about 1-2 minutes.//
  * Unplug the USB flash drive.
  * An archive with all the required logs has been created on the USB flash drive.


<code bash usb.autorun>
#!/bin/sh

# Version of the script
SCRIPT_VERSION=1.4

# Output directory for the result of the script
OUTPUT_DIR="`dirname $0`"

# Execution's environment of the script
SCRIPT_ENV=unknown

# PLATFORM: IBTS, IFEMTO, ISTATION, IFEVO
PLATFORM=unknown
# Serial number of the product: 39020003
SERIAL=unknown
# ETH0 MAC: 70:76:ff:01:77:71 => 7771
SHORT_MAC=unknown

# LED paths
SYSLEDPATH_4_0_IBTS="/sys/class/leds/yellow:debug"
SYSLEDPATH_4_0_IFEMTO="/sys/class/leds/led1:red:power"
SYSLEDPATH="/sys/class/leds/status"
LED_PATH="/do/not/exist"

# Script arguments
LEDS_MANAGEMENT=true
EXEC_COMMANDS=true

#
# Print the version of the script
#
show_version() {
        echo "Script version: ${SCRIPT_VERSION}"
}


#
# Show help
#
show_help()
{
        show_version
        echo -e "Usage: $0 [-h|--help] [-l|--ignore-leds] [-o|--output-dir <directory>]
\t-h|--help: this help
\t-v|--version: version of the script
\t-l|--ignore-leds: this script does not manage the leds (default: autodetection)
\t-o|--output-dir <directory>: force the output <directory> to use for extraction (default: current directory)
\t-c|--disable-commands: disable diagnosis commands (default: executed)
\t-m|--minimal: get minimal logs (to lower generated file size)
"
}

#
# Compare 2 versions to know if inferior or superior
# Res: 0: $1 = $2
#      1: $1 > $2
#      2: $1 < $2
#
compare_versions () {
        if [[ $1 == $2 ]]
        then
                return 0
        fi
        local IFS=.
        local i ver1=($1) ver2=($2)
        # fill empty fields in ver1 with zeros
        for ((i=${#ver1[@]}; i<${#ver2[@]}; i++)); do
                ver1[i]=0
        done
        for ((i=0; i<${#ver1[@]}; i++)); do
                if [[ -z ${ver2[i]} ]]
                then
                        # fill empty fields in ver2 with zeros
                        ver2[i]=0
                fi
                if ((10#${ver1[i]} > 10#${ver2[i]}))
                then
                        return 1
                fi
                if ((10#${ver1[i]} < 10#${ver2[i]}))
                then
                        return 2
                fi
        done
        return 0
}


#
# Copy a file or directory if exists
# Arguments:
#            $1: file/directory to copy
#            $2: destination directory (create if necessary)
#
copy_if_exists(){
        local from="$1"
        local to="$2"

        if [ -e "${from}" ] ; then
                [ ! -d "${to}" ] && mkdir -p "${to}"
                cp -a "${from}" "${to}"
        fi
}


#
# Check KerOS/SPN version and stop if:
#  - KerOS < 4.0
#  - SPN < 2.1
#
check_fw_version(){
        # check if /etc/version exists
        if [ -a /etc/version ]; then
                FW_VERSION=$(cat /etc/version)

                # Detect type of firmware: KerOS or SPN
                if [[ "${FW_VERSION}" == *-spn-* ]]; then
                        # Check if SPN version is not inferior to 2.1, stop if it is the case
                        compare_versions ${FW_VERSION:0:3} "2.1"
                        if [ $? -eq 2 ]; then
                                echo "Your SPN firmware version ($FW_VERSION) is inferior to v2.1. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
                                exit
                        fi
                else
                        # Check if KerOS version is not inferior to 4.0, stop if it is the case
                        compare_versions ${FW_VERSION:0:3} "4.0"
                        if [ $? -eq 2 ]; then
                                echo "Your KerOS firmware version ($FW_VERSION) is inferior to v4. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
                                exit
                        fi
                fi
        else
                echo "Could not find /etc/version" >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
                exit
        fi
}


#
# Check platform we are running on
#
detect_platform() {
        PLATFORM="$(fslotpreg getplatform)"
        if [ $? -ne 0 ];
        then
                PLATFORM="NOT_SUPPORTED"
        fi
        echo "Platform detected: ${PLATFORM}"
}


#
# Get product serial number information: SERIAL and SHORT_MAC
#
fill_serials() {
        local reg=$(fslotpreg read GP1)
        if [ $? -eq 0 ]; then
                SERIAL=$(printf "%X\n" $reg)
        else
                SERIAL="unknown"
        fi
        if [ -f "/sys/class/net/eth0/address" ]; then
                SHORT_MAC=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}')
        elif [ -f "/sys/class/net/usb0/address" ]; then
                SHORT_MAC=$(cat /sys/class/net/usb0/address | awk -F: '{print $5$6}')
        else
                SHORT_MAC="XXXX"
        fi
}


#
# Detect the type of environment where the script is launched.
# Possible environments:
#   - udev
#   - shell
#
detect_env() {
        # Detection is only based on the presence or not of the variable DEVNAME
        if [ ! -z ${DEVNAME} ] ; then
                SCRIPT_ENV=udev
                exec > /dev/console 2>&1 # Output the script execution in debug console
        else
                SCRIPT_ENV=shell
                LEDS_MANAGEMENT=false
                OUTPUT_DIR=$(pwd)
        fi
        echo "Script environment: ${SCRIPT_ENV}"
}


#
# Detect type of leds depending of platform/firmware version
#
detect_leds() {
        if [ -e "${SYSLEDPATH}" ]; then
                LED_PATH="${SYSLEDPATH}"
        elif [ -e "${SYSLEDPATH_4_0_IBTS}" ]; then
                LED_PATH="${SYSLEDPATH_4_0_IBTS}"
        elif [ -e "${SYSLEDPATH_4_0_IFEMTO}" ]; then
                LED_PATH="${SYSLEDPATH_4_0_IFEMTO}"
        fi
}


#
# Set default led behavior
# $1: trigger
#
#    if trigger = timer : $2 = delay_on, $3 = delay_off
#
set_led() {
        trigger="${1}"
        on="$2"
        off="$3"

        if [ -e "${LED_PATH}" ]; then
                echo $trigger > "${LED_PATH}/trigger"
                case "$trigger" in
                        "timer")
                                echo ${on:=500} > "${LED_PATH}/delay_on"
                                echo ${off:=500} > "${LED_PATH}/delay_off"
                                ;;
                        "none")
                                echo ${on:=0} > "${LED_PATH}/brightness"
                                ;;
                        *)
                                # Nothing to do
                                ;;
                esac
        fi
}

get_nmea_frames() {
        gps_tty="$1"

        if ! [ -e "$gps_tty" ]; then
                echo "$gps_tty doesn't exist"
                return 1
        fi

        if [ -n "$(lsof | grep $(readlink $gps_tty))" ]; then
                echo "$gps_tty already used by $(lsof | grep $(readlink $gps_tty))"
                return 2
        else
                # Pseudo-term keep only FIRST frames so we keep last of these old frames and get few new frames
                # It will take 3 seconds
                timeout -t 3 cat $gps_tty > /tmp/nmea.log
                cat /tmp/nmea.log
        fi
}

#
# Execute some diagnosis commands (can take a long time)
#
execute_commands() {
        echo "Executing diagnosis commands (can take a long time)"

        echo -e "\tSystems commands"
                (
                echo -e "\n\n\n**************  fslotpreg getplatform  *****************"
                fslotpreg getplatform
                echo -e "\n\n\n**************  fslotpreg fslotpreg read GP1  *****************"
                fslotpreg read GP1
                echo -e "\n\n\n**************  date  *****************"
                date
                echo -e "\n\n\n**************  uptime  *****************"
                uptime
                echo -e "\n\n\n**************  ps ww *****************"
                ps ww
                echo -e "\n\n\n**************  opkg status  *****************"
                opkg status
                echo -e "\n\n\n**************  monit status  *****************"
                monit status
                echo -e "\n\n\n**************  mount  *****************"
                mount
                echo -e "\n\n\n**************  OpenVPN Certificate expiration date  *****************"
                compare_versions ${FW_VERSION:0:3} "5.0"
                ret=$?
                if [ $ret -eq 1 ]; then
                        if [ -f "/etc/openvpn/bscc.p12" ]; then
                                openssl pkcs12 -info -in /etc/openvpn/bscc.p12 -nokeys -password file:/etc/openvpn/bscc.password| openssl x509 -noout -enddate
                        fi
                elif [ $ret -eq 2 ]; then
                        grep -iA3 "Validity" /.update/update.log | tail -3 | grep -i "not after" | sed 's/^[ \t]*//'
                fi
                echo -e "\n\n\n**************  Disk info (df -h)  *****************"
                df -h
                echo -e "\n\n\n**************  Log Size (du -sh /var/log)  *****************"
                du -sh /var/log
        ) >> ${tmpdir}/cmd_results.txt

        echo -e "\tNetwork commands"
        (
                echo -e "\n\n\n**************  ifconfig -a  *****************"
                ifconfig -a
                echo -e "\n\n\n**************  gsmdiag.py  *****************"
                gsmdiag.py ; cat /tmp/gsmdiag.txt
                echo -e "\n\n\n**************  route  *****************"
                route -n
                echo -e "\n\n\n**************  iptables -L  *****************"
                iptables -L
                echo -e "\n\n\n**************  connmanctl technologies  *****************"
                connmanctl technologies
                echo -e "\n\n\n**************  services  *****************"
                connmanctl services

                if [ -e /dev/cdc-wdm1 ]; then
                        echo -e "\n\n\n**************  qmicli get mode  *****************"
                        qmicli -d /dev/cdc-wdm1 --nas-get-system-selection-preference
                fi
        ) >> ${tmpdir}/network_result.txt

        get_nmea_frames /dev/nmea3 > ${tmpdir}/nmea.log
        [ $? -ne 0 ] && get_nmea_frames /dev/nmea2 >> ${tmpdir}/nmea.log

        # Overlay modifications
        echo -e "\tOverlay commands disabled (KEROS-1907)"
#       overlay diff -l /.rootfs.ro -u /user/.rootfs_upper/ >> ${tmpdir}/overlay_status.txt << EOF
#yes
#EOF

        echo "End of diagnosis commands"
}


#
# Get all the logs, configuration files and commands and save it
# in the resquested output directory.
# Arguments:
#            $1: output directory
#
get_logs()
{
        local output_dir="$1"
        local option="$2"
        local name="Logs_${SERIAL}_${SHORT_MAC}_$(date "+%Y%m%d-%H%M%S")"
        if [ "${option}" = "minimal" ]; then
                name="${name}_minimal"
        fi
        local tmpdir="/user/${name}"

        # tmp files
        echo "Extracting tmp files"
        copy_if_exists /tmp/sys_startup_status.json ${tmpdir}/tmp/
        copy_if_exists /tmp/board_info.json ${tmpdir}/tmp/
        copy_if_exists /tmp/calib_loraloc.json ${tmpdir}/tmp/
        copy_if_exists /tmp/calib_rf.json ${tmpdir}/tmp/

        # network configuration
        echo "Extracting network configuration"
        if [ "${option}" = "minimal" ]; then
                copy_if_exists /etc/network/connman/ ${tmpdir}/etc/network/
                copy_if_exists /etc/network/ofono/ ${tmpdir}/etc/network/
                copy_if_exists /etc/network/openvpn/ ${tmpdir}/etc/network/
                cp /etc/network/*.conf ${tmpdir}/etc/network/
        else
                copy_if_exists /etc/network/ ${tmpdir}/etc/
        fi
        copy_if_exists /etc/firewall.d/ ${tmpdir}/etc/

        # init configuration
        if [ "${option}" != "minimal" ]; then
                echo "Extracting init.d and rc*.d files"
                copy_if_exists /etc/init.d/ ${tmpdir}/etc/
                for filename in /etc/rc*.d; do
                        copy_if_exists ${filename} ${tmpdir}/etc/
                done
        fi

        # CPF (lorad + lorafwd) configuration
        echo "Extracting CPF configuration (if any)"
        if [ "${option}" = "minimal" ]; then
                copy_if_exists /user/etc/lorad/lorad.json ${tmpdir}/user/etc/lorad/
                copy_if_exists /etc/lorad/lorad.json ${tmpdir}/etc/lorad/
        else
                for lorad_dir in /user/etc/lorad /etc/lorad; do
                        if [ -d ${lorad_dir} ] ; then
                                echo -e "\tExtracting lorad configuration"
                                copy_if_exists ${lorad_dir} ${tmpdir}${lorad_dir%/*}
                        fi
                done
        fi
        copy_if_exists /etc/default/lorad ${tmpdir}/etc/default/

        echo -e "\tExtracting lorafwd configuration"
        if [ "${option}" = "minimal" ]; then
                for filename in /etc/lorafwd/*.toml; do
                        copy_if_exists ${filename} ${tmpdir}/etc/lorafwd/
                done
        else
                for lorafwd_dir in /user/etc/lorafwd /etc/lorafwd; do
                        if [ -d ${lorafwd_dir} ] ; then
                                copy_if_exists ${lorafwd_dir} ${tmpdir}${lorafwd_dir%/*}
                        fi
                done
        fi
        copy_if_exists /etc/lorafwd.toml ${tmpdir}/etc/
        copy_if_exists /etc/default/lorafwd ${tmpdir}/etc/default/

        # WMC configuration and logs
        echo "Extracting WMC configuration and logs (if any)"
        if [ -d /user/bscc ] ; then
                echo -e "\tExtracting BSCC configuration and logs"
                copy_if_exists /user/bscc/var.xml ${tmpdir}/user/bscc/
                copy_if_exists /user/bscc/traces/ ${tmpdir}/user/bscc/
        fi
        if [ -d /user/snmp ] ; then
                echo -e "\tExtracting SNMP configuration and logs"
                copy_if_exists /user/snmp/snmpd.conf ${tmpdir}/user/snmp/
                copy_if_exists /user/snmp/traces/ ${tmpdir}/user/snmp/
        fi
        copy_if_exists /etc/default/bscc ${tmpdir}/etc/default/
        copy_if_exists /etc/default/lora-snmp ${tmpdir}/etc/default/
        copy_if_exists /etc/snmp/snmpd.conf ${tmpdir}/etc/snmp/

        # basic station configuration and logs
        echo "Extracting basic station configuration and logs (if any)"
        if [ -d /user/basic_station ] ; then
                echo -e "\tExtracting Basic Station configuration and logs"
                copy_if_exists /user/basic_station/etc/ ${tmpdir}/user/basic_station/
                copy_if_exists /user/basic_station/log/ ${tmpdir}/user/basic_station/
        fi
        copy_if_exists /etc/station/ ${tmpdir}/etc/

        # miscellaneous configuration and logs
        echo "Extracting miscellaneous configuration and logs"
        copy_if_exists /etc/default/ ${tmpdir}/etc/
        copy_if_exists /etc/openvpn/ ${tmpdir}/etc/
        copy_if_exists /.update/update.log ${tmpdir}/
        copy_if_exists /user/initrd/initrd.log ${tmpdir}/user/initrd
        copy_if_exists /user/.update ${tmpdir}/user/
        copy_if_exists /user/.upgrade.log ${tmpdir}/user/
        copy_if_exists /user/.sysupgrade.log ${tmpdir}/user/
        copy_if_exists /etc/version ${tmpdir}/etc/
        copy_if_exists /etc/sysupgrade.conf ${tmpdir}/etc/
        copy_if_exists /etc/sysupgrade.d/ ${tmpdir}/etc/
        copy_if_exists /etc/.squash_layer_version ${tmpdir}/etc/
        echo ${SCRIPT_VERSION} > ${tmpdir}/script_version

        if ${EXEC_COMMANDS} ; then
                execute_commands
        fi

        # Finish by system logs to prevent space errors
        echo "Extracting system logs"
        if [ "${option}" = "minimal" ]; then
                copy_if_exists /var/log/messages ${tmpdir}/var/log/
                copy_if_exists /var/log/dmesg ${tmpdir}/var/log/
                for filename in /var/log/*.1; do
                        copy_if_exists ${filename} ${tmpdir}/var/log/
                done
                for filename in /var/log/*.log; do
                        copy_if_exists ${filename} ${tmpdir}/var/log/
                done
        else
                copy_if_exists /var/log/ ${tmpdir}/var/
        fi

        echo "Saving all data in ${output_dir}/${name}.tar.gz"
        tar -C /user -czf ${output_dir}/${name}.tar.gz ${name}
        echo "Done"
        sync

        rm -rf $tmpdir
}


#####################
#     MAIN
#####################

# Detect the environment of execution of the script
detect_env

# Manage scripts arguments
while [[ $# -gt 0 ]]
do
key="$1"

        case $key in
                -l|--ignore-leds)
                LEDS_MANAGEMENT=false
                shift
                ;;
                -o|--output-dir)
                OUTPUT_DIR="$2"
                shift 2
                ;;
                -c|--disable-commands)
                EXEC_COMMANDS=false
                shift
                ;;
                -h|--help)
                show_help
                shift
                exit 1
                ;;
                -v|--version)
                show_version
                shift
                exit 1
                ;;
                -m|--minimal)
                OPTION="minimal"
                shift
                ;;
                *)
                shift
                ;;
        esac
done

# Check if firmware version is compatible
check_fw_version

# Detect type of platform
detect_platform

# Fill SERIAL and SHORT_MAC variables
fill_serials

# Set led On
if ${LEDS_MANAGEMENT} ; then
        detect_leds
        set_led none 1
fi

# Get logs
get_logs ${OUTPUT_DIR} ${OPTION}

# End of script, manage user's notifications
if ${LEDS_MANAGEMENT} ; then
        # Prepare reboot after USB unplug
        cat > /tmp/usbkey_remove << EOF
# Switch off Led
echo "none" > ${LED_PATH}/trigger
EOF

        # All is right, make Led blinking fast to notify User
        set_led timer 100 100
fi
</code>

++++

===== Helium =====
This section contains some tips to do the first level of analysis (support level 1) on Helium gateways.\\
It will help you to investigate before escalate to Kerlink support (level 2).


**Please note that there is an existing open wiki for Helium users: [[https://wikikerlink.fr/helium|Kerlink Helium Hotspot Wiki]]**\\
  * First step is to follow [[https://wikikerlink.fr/helium/doku.php?id=home#troubleshooting_guide|Troubleshooting guide]] chapter which contains:
        * [[https://wikikerlink.fr/helium/doku.php?id=home#self_diagnostic|Self diagnostic]] section
        * [[https://wikikerlink.fr/helium/doku.php?id=home#useful_links|Useful links]] section

  * Second step, if necessary, is to follow the table below:

^ Source of info ^ KPI ^ Value ^ Action ^
^ [[https://helium-onboarding.kerlink.com/|Onboarding dashboard]] | Last seen | More than 30m ago | Gateway is offline. Ask customer to check power/internet. |
| ::: | ::: | Less than 30m ago | Gateway is online. Check next KPI. |
| ::: | Block height | = 0 | Gateway is downloading a snapshot. |
| ::: | ::: | = 1 | Gateway is loading the downloaded snapshot. |
| ::: | ::: | Else | KPI is ok, check next KPI. |
| ::: | Block age | > 30 mn | Gateway is syncing, check again in one hour. If it persists, you can open a ticket to Kerlink Support. |
| ::: | ::: | < 30 mn | Gateway is synced. Check next KPI. |
^ [[https://explorer.helium.com//|Helium explorer]] | 24h/14d earnings | > 0 HNT | Seems OK. Confirm with other Explorer KPIs. |
| ::: | Last activity date | < 3h | Seems OK. Confirm with other Explorer KPIs. |
| ::: | Is there a constructed challenge in the past 24h? | Yes | Seems OK. Confirm with other Explorer KPIs. |
| ::: | ::: | No | When gateway is synced, it should challenge. You can open a ticket to Kerlink Support. |
| ::: | Is there a challenged beaconer in the past 24h? | Yes | Seems OK. Confirm with other Explorer KPIs. |
| ::: | ::: | No | Gateway might be relayed. If not, you can open a ticket to Kerlink Support. |
| ::: | Is there a witnessed beacon in the past 8-24h? | Value depends on density around miner | Seems OK. Confirm with other Explorer KPIs. |
| ::: | ::: | No | You can open a ticket to Kerlink Support. |
| ::: | Is there a “broadcasted beacon” in the past 5d? | Yes | Seems OK. Confirm with other Explorer KPIs. |
| ::: | ::: | No | This is random but it seems odd. You can open a ticket to Kerlink Support. |

==== Gateways connected to a WMC====
For gateways connected to a WMC, you can use the Remote Command: Remote Access > Command\\
{{:images:remote_command_wanesy_management_center.png?800|}}
<note important>** Due its cryptographic key and to ensure Kerlink Helium Hotspot security, no SSH access can be given to those gateways. \\     **</note>

You can use commands below:
^ Command ^ Awaited result ^ 
| opkg list | {{:images:opkg.png?400|}} |
| monit status miner | {{:images:monit_status_miner.png?400|}} |Status = OK | 
| monit status lorafwd_helium | {{:images:monit_status_lorafwd_helium.png?400|}} | Status = OK |
| /etc/init.d/miner status | {{:images:miner_status.png?400|}} | Check Diff |

If the result is different than the awaited result, you can open a ticket to Kerlink Support.
Wirnet™ i-series

User Tools

Site Tools

Page Tools
