====== Troubleshoot the gateway ======
===== Useful logs =====
When configuring the gateway or when something unusual happens, most of the time, to understand the behaviour of the gateway it is advised to consult the logs.
Here is the list of the most useful logs:
* ''/var/log/messages*'': These files are the main logs, they gather the traces of the firmware. They contain the boot traces, the network events, the hardware events, and many other traces.
* ''/var/log/lora.log*'': Theses files contain the traces of the common packet forwarder (lorad and lorafwd). If you encounter LoRa issues, this is a good way to start. The verbosity of this log can be increased/decreased. Refer to the [[ wiki:lora:cpf#log_management |log management]] section.
* ''/tmp/board_info.json'': This file contains information about the hardware of the gateway. It is generated at each boot time.\\
* ''/tmp/calib_loraloc.json'' (**for Wirnet iBTS**): This file contains the calibration values specific to this gateway. It is generated at boot time.
* ''/tmp/calib_rf.json'' (**for Wirnet iFemtoCell, Wirnet iFemtoCell-evolution and Wirnet iStation**): This file contains the calibration values specific to this gateway. It is generated at boot time.
* ''/dev/nmea*'' (**Wirnet iBTS and Wirnet iStation**): These devices receive NMEA frames (GPS) every seconds. It can be monitored with the ''cat'' command.
===== Useful commands =====
Since Kerlink gateways use a Unix system, most current commands are available (''ifconfig'', ''route'', ''ps'', ''iptables'', ...).
**KerOS specific commands:**
* ''gsmdiag.py'': generates a status of the modem.
* ''connmanctl'': used to configure/monitor the network manager.
* ''opkg list-installed'': lists the installed packages.
* ''monit status'': displays the status of the monitored processes.
===== Bootcause =====
The ''bootcount'' value is counting the number of reboots since the last power cycle.
At each power loss the counter is reset to 0. The maximum value of this counter is 65535.\\
The ''bootcount'' value can be checked in the file ''/tmp/board_info.json'':
# grep bootcount /tmp/board_info.json
"bootcount": 1,
When a gateway is started/restarted, U-Boot displays the boot cause.\\
The possible reasons are:
* POR: Power On Reset (happens when the gateway is powered or after a powerloss).
* WDOG: Watchdog.
* SW: Software Reset.
* HW: Physical reset (RESET button).
* UNKNOWN: unknown cause.
This information is transmitted to initrd and to applications with the kernel argument « bootcause ». \\
The information is then provided in the ''bootcause'' parameter in the file **''/proc/cmdline''**. \\
Example:
# cat /proc/cmdline
console=ttymxc0,115200 bootmode=nominal bootcause=SW bootcount=5 bootfail=0
===== Contacting Kerlink's support team =====
In case you need assistance to troubleshoot your gateway, Kerlink's support team will help you if:
* The gateway is still under warranty.
* The gateway has been bought directly from Kerlink.
If the gateway does not satisfy this requirement, please contact [[sales@kerlink.fr]]. Our Sales team will make you an offer to get assistance.
Otherwise, contact [[support@kerlink.fr]] to get assistance. Our support team will open you an account in Kerlink's ticketing tool.
When contacting the support team, to improve and speed up the resolution of your issue, please provide them:
* The serial number of your gateway(s):
* For Wirnet iBTS, the serial number is written on a sticker inside the case with "Wirnet iBTS" written on it.
* For Wirnet iFemtoCell, the serial number is written on the rear sticker (it is named "Final product ID").
* For Wirnet iStation, the serial number is written on the external sticker (it is named "Product ID").
* For Wirnet iFemtoCell-evolution, the serial number is written on rear sticker (it is named "Product ID").
* A precise description of your problem. "Gateway not working" is not enough to investigate.
* The list of the actions you took to troubleshoot/investigate your problem.
* The date of your problem.
* The version of your firmware.
* The logs of your gateway (refer to the automatic log gathering section). Without these logs, it is difficult to investigate. If you are able to reproduce the problem, make sure to reproduce it and gather the logs afterwards.
* Any information that would help us understand the context. For example, if the gateway is on the field or in a lab, if a third party program is used, if the gateway was previously working, etc.
===== Automatic log gathering =====
To simplify the log gathering process, two scripts are available. These scripts generate an archive containing many logs and commands results.
++++Gathering logs from WebUI since release 5.9|
[[wiki:systeme_mana:webui#gateway_administration_menu|Get logs from gateway]]
++++
++++Gathering logs from shell since release 4.2|
To gather the logs from shell:
* Execute the command ''get_logs''. //The execution of the script takes less than 1 minute.//
* It will generate an archive in the same directory you executed the script (example: ''Logs_2E0605F5_7b26_20200124-095206.tar.gz'')
* Retrieve this archive (use whatever method you prefer: FileZilla, scp, ...).
**To use only in case of speed or data restriction**, there is a new option to gather lighter logs: ''get_logs -m'' or ''get_logs --minimal''. \\
**Please note that it will only gather latest logs** (.log & .log.1 files). \\
It will generate an light archive recognizable by its name: _minimal.tar.gz (example: ''Logs_3903002E_0091_20230419-134556_minimal.tar.gz'')
++++
++++Gathering logs from shell on older releases|
To gather the logs from shell:
* Upload the script bellow (use whatever method you prefer: FileZilla, scp, copy/paste, ...) in a folder of your gateway and go to this folder.
* Convert the file to Unix text file format using ''dos2unix get_logs.sh''.
* Add execution rights to the script using ''chmod 755 get_logs.sh''.
* Execute the script using ''./get_logs.sh''. //The execution of the script takes less than 1 minute.//
* It will generate an archive in the same directory you executed the script. (example: ''Logs_2E0605F5_7b26_20200124-095206.tar.gz'')
* Retrieve this archive (use whatever method you prefer: FileZilla, scp, ...).
#!/bin/sh
# Version of the script
SCRIPT_VERSION=1.3
# Output directory for the result of the script
OUTPUT_DIR="`dirname $0`"
# Execution's environment of the script
SCRIPT_ENV=unknown
# PLATFORM: IBTS, IFEMTO, ISTATION, IFEVO
PLATFORM=unknown
# Serial number of the product: 39020003
SERIAL=unknown
# ETH0 MAC: 70:76:ff:01:77:71 => 7771
SHORT_MAC=unknown
# LED paths
SYSLEDPATH_4_0_IBTS="/sys/class/leds/yellow:debug"
SYSLEDPATH_4_0_IFEMTO="/sys/class/leds/led1:red:power"
SYSLEDPATH="/sys/class/leds/status"
LED_PATH="/do/not/exist"
# Script arguments
LEDS_MANAGEMENT=true
EXEC_COMMANDS=true
#
# Print the version of the script
#
show_version() {
echo "Script version: ${SCRIPT_VERSION}"
}
#
# Show help
#
show_help()
{
show_version
echo -e "Usage: $0 [-h|--help] [-l|--ignore-leds] [-o|--output-dir ]
\t-h|--help: this help
\t-v|--version: version of the script
\t-l|--ignore-leds: this script does not manage the leds (default: autodetection)
\t-o|--output-dir : force the output to use for extraction (default: current directory)
\t-c|--disable-commands: disable diagnosis commands (default: executed)
"
}
#
# Compare 2 versions to know if inferior or superior
# Res: 0: $1 = $2
# 1: $1 > $2
# 2: $1 < $2
#
compare_versions () {
if [[ $1 == $2 ]]
then
return 0
fi
local IFS=.
local i ver1=($1) ver2=($2)
# fill empty fields in ver1 with zeros
for ((i=${#ver1[@]}; i<${#ver2[@]}; i++))
do
ver1[i]=0
done
for ((i=0; i<${#ver1[@]}; i++))
do
if [[ -z ${ver2[i]} ]]
then
# fill empty fields in ver2 with zeros
ver2[i]=0
fi
if ((10#${ver1[i]} > 10#${ver2[i]}))
then
return 1
fi
if ((10#${ver1[i]} < 10#${ver2[i]}))
then
return 2
fi
done
return 0
}
#
# Copy a file or directory if exists
# Arguments:
# $1: file/directory to copy
# $2: destination directory (create if necessary)
#
copy_if_exists(){
local from="$1"
local to="$2"
if [ -e "${from}" ] ; then
[ ! -d "${to}" ] && mkdir -p "${to}"
cp -a "${from}" "${to}"
fi
}
#
# Check KerOS/SPN version and stop if:
# - KerOS < 4.0
# - SPN < 2.1
#
check_fw_version(){
# check if /etc/version exists
if [ -a /etc/version ]; then
FW_VERSION=$(cat /etc/version)
# Detect type of firmware: KerOS or SPN
if [[ "${FW_VERSION}" == *-spn-* ]]; then
# Check if SPN version is not inferior to 2.1, stop if it is the case
compare_versions ${FW_VERSION:0:3} "2.1"
if [ $? -eq 2 ]; then
echo "Your SPN firmware version ($FW_VERSION) is inferior to v2.1. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
exit
fi
else
# Check if KerOS version is not inferior to 4.0, stop if it is the case
compare_versions ${FW_VERSION:0:3} "4.0"
if [ $? -eq 2 ]; then
echo "Your KerOS firmware version ($FW_VERSION) is inferior to v4. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
exit
fi
fi
else
echo "Could not find /etc/version" >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
exit
fi
}
#
# Check platform we are running on
#
detect_platform() {
PLATFORM="$(fslotpreg getplatform)"
if [ $? -ne 0 ];
then
PLATFORM="NOT_SUPPORTED"
fi
echo "Platform detected: ${PLATFORM}"
}
#
# Get product serial number information: SERIAL and SHORT_MAC
#
fill_serials() {
local reg=$(fslotpreg read GP1)
if [ $? -eq 0 ]; then
SERIAL=$(printf "%X\n" $reg)
else
SERIAL="unknown"
fi
SHORT_MAC=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}')
}
#
# Detect the type of environment where the script is launched.
# Possible environments:
# - udev
# - shell
#
detect_env() {
# Detection is only based on the presence or not of the variable DEVNAME
if [ ! -z ${DEVNAME} ] ; then
SCRIPT_ENV=udev
exec > /dev/console 2>&1 # Output the script execution in debug console
else
SCRIPT_ENV=shell
LEDS_MANAGEMENT=false
OUTPUT_DIR=$(pwd)
fi
echo "Script environment: ${SCRIPT_ENV}"
}
#
# Detect type of leds depending of platform/firmware version
#
detect_leds() {
if [ -e "${SYSLEDPATH}" ]; then
LED_PATH="${SYSLEDPATH}"
elif [ -e "${SYSLEDPATH_4_0_IBTS}" ]; then
LED_PATH="${SYSLEDPATH_4_0_IBTS}"
elif [ -e "${SYSLEDPATH_4_0_IFEMTO}" ]; then
LED_PATH="${SYSLEDPATH_4_0_IFEMTO}"
fi
}
#
# Set default led behavior
# $1: trigger
#
# if trigger = timer : $2 = delay_on, $3 = delay_off
#
set_led() {
trigger="${1}"
on="$2"
off="$3"
if [ -e "${LED_PATH}" ]; then
echo $trigger > "${LED_PATH}/trigger"
case "$trigger" in
"timer")
echo ${on:=500} > "${LED_PATH}/delay_on"
echo ${off:=500} > "${LED_PATH}/delay_off"
;;
"none")
echo ${on:=0} > "${LED_PATH}/brightness"
;;
*)
# Nothing to do
;;
esac
fi
}
get_nmea_frames() {
gps_tty="$1"
if ! [ -e "$gps_tty" ]; then
echo "$gps_tty doesn't exist"
return 1
fi
if [ -n "$(lsof | grep $(readlink $gps_tty))" ]; then
echo "$gps_tty already used by $(lsof | grep $(readlink $gps_tty))"
return 2
else
# Pseudo-term keep only FIRST frames so we keep last of these old frames and get few new frames
# It will take 3 seconds
timeout -t 3 cat $gps_tty > /tmp/nmea.log
cat /tmp/nmea.log
fi
}
#
# Execute some diagnosis commands (can take a long time)
#
execute_commands() {
echo "Executing diagnosis commands (can take a long time)"
echo -e "\tSystems commands"
(
echo -e "\n\n\n************** fslotpreg getplatform *****************"
fslotpreg getplatform
echo -e "\n\n\n************** fslotpreg fslotpreg read GP1 *****************"
fslotpreg read GP1
echo -e "\n\n\n************** date *****************"
date
echo -e "\n\n\n************** uptime *****************"
uptime
echo -e "\n\n\n************** ps ww *****************"
ps ww
echo -e "\n\n\n************** opkg status *****************"
opkg status
echo -e "\n\n\n************** monit status *****************"
monit status
echo -e "\n\n\n************** mount *****************"
mount
) >> ${tmpdir}/cmd_results.txt
echo -e "\tNetwork commands"
(
echo -e "\n\n\n************** ifconfig -a *****************"
ifconfig -a
echo -e "\n\n\n************** gsmdiag.py *****************"
gsmdiag.py ; cat /tmp/gsmdiag.txt
echo -e "\n\n\n************** route *****************"
route -n
echo -e "\n\n\n************** iptables -L *****************"
iptables -L
echo -e "\n\n\n************** connmanctl technologies *****************"
connmanctl technologies
echo -e "\n\n\n************** services *****************"
connmanctl services
if [ -e /dev/cdc-wdm1 ]; then
echo -e "\n\n\n************** qmicli get mode *****************"
qmicli -d /dev/cdc-wdm1 --nas-get-system-selection-preference
fi
) >> ${tmpdir}/network_result.txt
get_nmea_frames /dev/nmea3 > ${tmpdir}/nmea.log
[ $? -ne 0 ] && get_nmea_frames /dev/nmea2 >> ${tmpdir}/nmea.log
# Overlay modifications
echo -e "\tOverlay commands"
overlay diff -l /.rootfs.ro -u /user/.rootfs_upper/ >> ${tmpdir}/overlay_status.txt << EOF
yes
EOF
echo "End of diagnosis commands"
}
#
# Get all the logs, configuration files and commands and save it
# in the resquested output directory.
# Arguments:
# $1: output directory
#
get_logs()
{
local output_dir="$1"
local name="Logs_${SERIAL}_${SHORT_MAC}_$(date "+%Y%m%d-%H%M%S")"
local tmpdir="/user/${name}"
# tmp files
echo "Extracting tmp files"
copy_if_exists /tmp/sys_startup_status.json ${tmpdir}/tmp/
copy_if_exists /tmp/board_info.json ${tmpdir}/tmp/
copy_if_exists /tmp/calib_loraloc.json ${tmpdir}/tmp/
copy_if_exists /tmp/calib_rf.json ${tmpdir}/tmp/
# network configuration
echo "Extracting network configuration"
copy_if_exists /etc/network/ ${tmpdir}/etc/
copy_if_exists /etc/firewall.d/ ${tmpdir}/etc/
# CPF (lorad + lorafwd) configuration
echo "Extracting CPF configuration (if any)"
for lorad_dir in /user/etc/lorad /etc/lorad
do
if [ -d ${lorad_dir} ] ; then
echo -e "\tExtracting lorad configuration"
copy_if_exists /etc/default/lorad ${tmpdir}/etc/default/
copy_if_exists ${lorad_dir} ${tmpdir}${lorad_dir%/*}
fi
done
for lorafwd_dir in /user/etc/lorafwd /etc/lorafwd
do
if [ -d ${lorafwd_dir} ] ; then
echo -e "\tExtracting lorafwd configuration"
copy_if_exists /etc/default/lorafwd ${tmpdir}/etc/default/
copy_if_exists ${lorafwd_dir} ${tmpdir}${lorafwd_dir%/*}
fi
done
# WMC configuration and logs
echo "Extracting WMC configuration and logs (if any)"
if [ -d /user/bscc ] ; then
echo -e "\tExtracting BSCC configuration and logs"
copy_if_exists /user/bscc/var.xml ${tmpdir}/user/bscc/
copy_if_exists /user/bscc/traces/ ${tmpdir}/user/bscc/
fi
if [ -d /user/snmp ] ; then
echo -e "\tExtracting SNMP configuration and logs"
copy_if_exists /user/snmp/snmpd.conf ${tmpdir}/user/snmp/
copy_if_exists /user/snmp/traces/ ${tmpdir}/user/snmp/
fi
# miscellaneous configuration and logs
echo "Extracting miscellaneous configuration and logs"
copy_if_exists /etc/rcU.d/ ${tmpdir}/etc/
copy_if_exists /.update/update.log ${tmpdir}/
copy_if_exists /user/.update ${tmpdir}/user/
copy_if_exists /user/.upgrade.log ${tmpdir}/user/
copy_if_exists /user/.sysupgrade.log ${tmpdir}/user/
copy_if_exists /etc/version ${tmpdir}/etc/
copy_if_exists /etc/sysupgrade.conf ${tmpdir}/etc/
copy_if_exists /etc/sysupgrade.d/ ${tmpdir}/etc/
copy_if_exists /etc/.squash_layer_version ${tmpdir}/etc/
echo ${SCRIPT_VERSION} > ${tmpdir}/script_version
if ${EXEC_COMMANDS} ; then
execute_commands
fi
# Finish by system logs to prevent space errors
echo "Extracting system logs"
copy_if_exists /var/log/ ${tmpdir}/var/
echo "Saving all data in ${output_dir}/${name}.tar.gz"
tar -C /user -czf ${output_dir}/${name}.tar.gz ${name}
echo "Done"
sync
rm -rf $tmpdir
}
#####################
# MAIN
#####################
# Detect the environment of execution of the script
detect_env
# Manage scripts arguments
while [[ $# -gt 0 ]]
do
key="$1"
case $key in
-l|--ignore-leds)
LEDS_MANAGEMENT=false
shift
;;
-o|--output-dir)
OUTPUT_DIR="$2"
shift 2
;;
-c|--disable-commands)
EXEC_COMMANDS=false
shift
;;
-h|--help)
show_help
shift
exit 1
;;
-v|--version)
show_version
shift
exit 1
;;
*)
shift
;;
esac
done
# Check if firmware version is compatible
check_fw_version
# Detect type of platform
detect_platform
# Fill SERIAL and SHORT_MAC variables
fill_serials
# Set led On
if ${LEDS_MANAGEMENT} ; then
detect_leds
set_led none 1
fi
# Get logs
get_logs ${OUTPUT_DIR}
# End of script, manage user's notifications
if ${LEDS_MANAGEMENT} ; then
# Prepare reboot after USB unplug
cat > /tmp/usbkey_remove << EOF
# Switch off Led
echo "none" > ${LED_PATH}/trigger
EOF
# All is right, make Led blinking fast to notify User
set_led timer 100 100
fi
++++
++++Gathering logs from USB (no commands required)|
To gather logs from a USB without typing any command:
* Download the following script and copy it on a USB flash drive.
* Create a ''usbkey.txt'' file on the USB flash drive as described [[wiki:keros_custo:sw_updates#specific_files|here]].
* Plug the USB flash drive into the USB port of the gateway.
* Wait for the yellow debug led fast blinking. //The execution of the script takes about 1-2 minutes.//
* Unplug the USB flash drive.
* An archive with all the required logs has been created on the USB flash drive.
#!/bin/sh
# Version of the script
SCRIPT_VERSION=1.4
# Output directory for the result of the script
OUTPUT_DIR="`dirname $0`"
# Execution's environment of the script
SCRIPT_ENV=unknown
# PLATFORM: IBTS, IFEMTO, ISTATION, IFEVO
PLATFORM=unknown
# Serial number of the product: 39020003
SERIAL=unknown
# ETH0 MAC: 70:76:ff:01:77:71 => 7771
SHORT_MAC=unknown
# LED paths
SYSLEDPATH_4_0_IBTS="/sys/class/leds/yellow:debug"
SYSLEDPATH_4_0_IFEMTO="/sys/class/leds/led1:red:power"
SYSLEDPATH="/sys/class/leds/status"
LED_PATH="/do/not/exist"
# Script arguments
LEDS_MANAGEMENT=true
EXEC_COMMANDS=true
#
# Print the version of the script
#
show_version() {
echo "Script version: ${SCRIPT_VERSION}"
}
#
# Show help
#
show_help()
{
show_version
echo -e "Usage: $0 [-h|--help] [-l|--ignore-leds] [-o|--output-dir ]
\t-h|--help: this help
\t-v|--version: version of the script
\t-l|--ignore-leds: this script does not manage the leds (default: autodetection)
\t-o|--output-dir : force the output to use for extraction (default: current directory)
\t-c|--disable-commands: disable diagnosis commands (default: executed)
\t-m|--minimal: get minimal logs (to lower generated file size)
"
}
#
# Compare 2 versions to know if inferior or superior
# Res: 0: $1 = $2
# 1: $1 > $2
# 2: $1 < $2
#
compare_versions () {
if [[ $1 == $2 ]]
then
return 0
fi
local IFS=.
local i ver1=($1) ver2=($2)
# fill empty fields in ver1 with zeros
for ((i=${#ver1[@]}; i<${#ver2[@]}; i++)); do
ver1[i]=0
done
for ((i=0; i<${#ver1[@]}; i++)); do
if [[ -z ${ver2[i]} ]]
then
# fill empty fields in ver2 with zeros
ver2[i]=0
fi
if ((10#${ver1[i]} > 10#${ver2[i]}))
then
return 1
fi
if ((10#${ver1[i]} < 10#${ver2[i]}))
then
return 2
fi
done
return 0
}
#
# Copy a file or directory if exists
# Arguments:
# $1: file/directory to copy
# $2: destination directory (create if necessary)
#
copy_if_exists(){
local from="$1"
local to="$2"
if [ -e "${from}" ] ; then
[ ! -d "${to}" ] && mkdir -p "${to}"
cp -a "${from}" "${to}"
fi
}
#
# Check KerOS/SPN version and stop if:
# - KerOS < 4.0
# - SPN < 2.1
#
check_fw_version(){
# check if /etc/version exists
if [ -a /etc/version ]; then
FW_VERSION=$(cat /etc/version)
# Detect type of firmware: KerOS or SPN
if [[ "${FW_VERSION}" == *-spn-* ]]; then
# Check if SPN version is not inferior to 2.1, stop if it is the case
compare_versions ${FW_VERSION:0:3} "2.1"
if [ $? -eq 2 ]; then
echo "Your SPN firmware version ($FW_VERSION) is inferior to v2.1. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
exit
fi
else
# Check if KerOS version is not inferior to 4.0, stop if it is the case
compare_versions ${FW_VERSION:0:3} "4.0"
if [ $? -eq 2 ]; then
echo "Your KerOS firmware version ($FW_VERSION) is inferior to v4. Please use the adequate script." >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
exit
fi
fi
else
echo "Could not find /etc/version" >> ${OUTPUT_DIR}/WRONG_FIRMWARE_VERSION.txt
exit
fi
}
#
# Check platform we are running on
#
detect_platform() {
PLATFORM="$(fslotpreg getplatform)"
if [ $? -ne 0 ];
then
PLATFORM="NOT_SUPPORTED"
fi
echo "Platform detected: ${PLATFORM}"
}
#
# Get product serial number information: SERIAL and SHORT_MAC
#
fill_serials() {
local reg=$(fslotpreg read GP1)
if [ $? -eq 0 ]; then
SERIAL=$(printf "%X\n" $reg)
else
SERIAL="unknown"
fi
if [ -f "/sys/class/net/eth0/address" ]; then
SHORT_MAC=$(cat /sys/class/net/eth0/address | awk -F: '{print $5$6}')
elif [ -f "/sys/class/net/usb0/address" ]; then
SHORT_MAC=$(cat /sys/class/net/usb0/address | awk -F: '{print $5$6}')
else
SHORT_MAC="XXXX"
fi
}
#
# Detect the type of environment where the script is launched.
# Possible environments:
# - udev
# - shell
#
detect_env() {
# Detection is only based on the presence or not of the variable DEVNAME
if [ ! -z ${DEVNAME} ] ; then
SCRIPT_ENV=udev
exec > /dev/console 2>&1 # Output the script execution in debug console
else
SCRIPT_ENV=shell
LEDS_MANAGEMENT=false
OUTPUT_DIR=$(pwd)
fi
echo "Script environment: ${SCRIPT_ENV}"
}
#
# Detect type of leds depending of platform/firmware version
#
detect_leds() {
if [ -e "${SYSLEDPATH}" ]; then
LED_PATH="${SYSLEDPATH}"
elif [ -e "${SYSLEDPATH_4_0_IBTS}" ]; then
LED_PATH="${SYSLEDPATH_4_0_IBTS}"
elif [ -e "${SYSLEDPATH_4_0_IFEMTO}" ]; then
LED_PATH="${SYSLEDPATH_4_0_IFEMTO}"
fi
}
#
# Set default led behavior
# $1: trigger
#
# if trigger = timer : $2 = delay_on, $3 = delay_off
#
set_led() {
trigger="${1}"
on="$2"
off="$3"
if [ -e "${LED_PATH}" ]; then
echo $trigger > "${LED_PATH}/trigger"
case "$trigger" in
"timer")
echo ${on:=500} > "${LED_PATH}/delay_on"
echo ${off:=500} > "${LED_PATH}/delay_off"
;;
"none")
echo ${on:=0} > "${LED_PATH}/brightness"
;;
*)
# Nothing to do
;;
esac
fi
}
get_nmea_frames() {
gps_tty="$1"
if ! [ -e "$gps_tty" ]; then
echo "$gps_tty doesn't exist"
return 1
fi
if [ -n "$(lsof | grep $(readlink $gps_tty))" ]; then
echo "$gps_tty already used by $(lsof | grep $(readlink $gps_tty))"
return 2
else
# Pseudo-term keep only FIRST frames so we keep last of these old frames and get few new frames
# It will take 3 seconds
timeout -t 3 cat $gps_tty > /tmp/nmea.log
cat /tmp/nmea.log
fi
}
#
# Execute some diagnosis commands (can take a long time)
#
execute_commands() {
echo "Executing diagnosis commands (can take a long time)"
echo -e "\tSystems commands"
(
echo -e "\n\n\n************** fslotpreg getplatform *****************"
fslotpreg getplatform
echo -e "\n\n\n************** fslotpreg fslotpreg read GP1 *****************"
fslotpreg read GP1
echo -e "\n\n\n************** date *****************"
date
echo -e "\n\n\n************** uptime *****************"
uptime
echo -e "\n\n\n************** ps ww *****************"
ps ww
echo -e "\n\n\n************** opkg status *****************"
opkg status
echo -e "\n\n\n************** monit status *****************"
monit status
echo -e "\n\n\n************** mount *****************"
mount
echo -e "\n\n\n************** OpenVPN Certificate expiration date *****************"
compare_versions ${FW_VERSION:0:3} "5.0"
ret=$?
if [ $ret -eq 1 ]; then
if [ -f "/etc/openvpn/bscc.p12" ]; then
openssl pkcs12 -info -in /etc/openvpn/bscc.p12 -nokeys -password file:/etc/openvpn/bscc.password| openssl x509 -noout -enddate
fi
elif [ $ret -eq 2 ]; then
grep -iA3 "Validity" /.update/update.log | tail -3 | grep -i "not after" | sed 's/^[ \t]*//'
fi
echo -e "\n\n\n************** Disk info (df -h) *****************"
df -h
echo -e "\n\n\n************** Log Size (du -sh /var/log) *****************"
du -sh /var/log
) >> ${tmpdir}/cmd_results.txt
echo -e "\tNetwork commands"
(
echo -e "\n\n\n************** ifconfig -a *****************"
ifconfig -a
echo -e "\n\n\n************** gsmdiag.py *****************"
gsmdiag.py ; cat /tmp/gsmdiag.txt
echo -e "\n\n\n************** route *****************"
route -n
echo -e "\n\n\n************** iptables -L *****************"
iptables -L
echo -e "\n\n\n************** connmanctl technologies *****************"
connmanctl technologies
echo -e "\n\n\n************** services *****************"
connmanctl services
if [ -e /dev/cdc-wdm1 ]; then
echo -e "\n\n\n************** qmicli get mode *****************"
qmicli -d /dev/cdc-wdm1 --nas-get-system-selection-preference
fi
) >> ${tmpdir}/network_result.txt
get_nmea_frames /dev/nmea3 > ${tmpdir}/nmea.log
[ $? -ne 0 ] && get_nmea_frames /dev/nmea2 >> ${tmpdir}/nmea.log
# Overlay modifications
echo -e "\tOverlay commands disabled (KEROS-1907)"
# overlay diff -l /.rootfs.ro -u /user/.rootfs_upper/ >> ${tmpdir}/overlay_status.txt << EOF
#yes
#EOF
echo "End of diagnosis commands"
}
#
# Get all the logs, configuration files and commands and save it
# in the resquested output directory.
# Arguments:
# $1: output directory
#
get_logs()
{
local output_dir="$1"
local option="$2"
local name="Logs_${SERIAL}_${SHORT_MAC}_$(date "+%Y%m%d-%H%M%S")"
if [ "${option}" = "minimal" ]; then
name="${name}_minimal"
fi
local tmpdir="/user/${name}"
# tmp files
echo "Extracting tmp files"
copy_if_exists /tmp/sys_startup_status.json ${tmpdir}/tmp/
copy_if_exists /tmp/board_info.json ${tmpdir}/tmp/
copy_if_exists /tmp/calib_loraloc.json ${tmpdir}/tmp/
copy_if_exists /tmp/calib_rf.json ${tmpdir}/tmp/
# network configuration
echo "Extracting network configuration"
if [ "${option}" = "minimal" ]; then
copy_if_exists /etc/network/connman/ ${tmpdir}/etc/network/
copy_if_exists /etc/network/ofono/ ${tmpdir}/etc/network/
copy_if_exists /etc/network/openvpn/ ${tmpdir}/etc/network/
cp /etc/network/*.conf ${tmpdir}/etc/network/
else
copy_if_exists /etc/network/ ${tmpdir}/etc/
fi
copy_if_exists /etc/firewall.d/ ${tmpdir}/etc/
# init configuration
if [ "${option}" != "minimal" ]; then
echo "Extracting init.d and rc*.d files"
copy_if_exists /etc/init.d/ ${tmpdir}/etc/
for filename in /etc/rc*.d; do
copy_if_exists ${filename} ${tmpdir}/etc/
done
fi
# CPF (lorad + lorafwd) configuration
echo "Extracting CPF configuration (if any)"
if [ "${option}" = "minimal" ]; then
copy_if_exists /user/etc/lorad/lorad.json ${tmpdir}/user/etc/lorad/
copy_if_exists /etc/lorad/lorad.json ${tmpdir}/etc/lorad/
else
for lorad_dir in /user/etc/lorad /etc/lorad; do
if [ -d ${lorad_dir} ] ; then
echo -e "\tExtracting lorad configuration"
copy_if_exists ${lorad_dir} ${tmpdir}${lorad_dir%/*}
fi
done
fi
copy_if_exists /etc/default/lorad ${tmpdir}/etc/default/
echo -e "\tExtracting lorafwd configuration"
if [ "${option}" = "minimal" ]; then
for filename in /etc/lorafwd/*.toml; do
copy_if_exists ${filename} ${tmpdir}/etc/lorafwd/
done
else
for lorafwd_dir in /user/etc/lorafwd /etc/lorafwd; do
if [ -d ${lorafwd_dir} ] ; then
copy_if_exists ${lorafwd_dir} ${tmpdir}${lorafwd_dir%/*}
fi
done
fi
copy_if_exists /etc/lorafwd.toml ${tmpdir}/etc/
copy_if_exists /etc/default/lorafwd ${tmpdir}/etc/default/
# WMC configuration and logs
echo "Extracting WMC configuration and logs (if any)"
if [ -d /user/bscc ] ; then
echo -e "\tExtracting BSCC configuration and logs"
copy_if_exists /user/bscc/var.xml ${tmpdir}/user/bscc/
copy_if_exists /user/bscc/traces/ ${tmpdir}/user/bscc/
fi
if [ -d /user/snmp ] ; then
echo -e "\tExtracting SNMP configuration and logs"
copy_if_exists /user/snmp/snmpd.conf ${tmpdir}/user/snmp/
copy_if_exists /user/snmp/traces/ ${tmpdir}/user/snmp/
fi
copy_if_exists /etc/default/bscc ${tmpdir}/etc/default/
copy_if_exists /etc/default/lora-snmp ${tmpdir}/etc/default/
copy_if_exists /etc/snmp/snmpd.conf ${tmpdir}/etc/snmp/
# basic station configuration and logs
echo "Extracting basic station configuration and logs (if any)"
if [ -d /user/basic_station ] ; then
echo -e "\tExtracting Basic Station configuration and logs"
copy_if_exists /user/basic_station/etc/ ${tmpdir}/user/basic_station/
copy_if_exists /user/basic_station/log/ ${tmpdir}/user/basic_station/
fi
copy_if_exists /etc/station/ ${tmpdir}/etc/
# miscellaneous configuration and logs
echo "Extracting miscellaneous configuration and logs"
copy_if_exists /etc/default/ ${tmpdir}/etc/
copy_if_exists /etc/openvpn/ ${tmpdir}/etc/
copy_if_exists /.update/update.log ${tmpdir}/
copy_if_exists /user/initrd/initrd.log ${tmpdir}/user/initrd
copy_if_exists /user/.update ${tmpdir}/user/
copy_if_exists /user/.upgrade.log ${tmpdir}/user/
copy_if_exists /user/.sysupgrade.log ${tmpdir}/user/
copy_if_exists /etc/version ${tmpdir}/etc/
copy_if_exists /etc/sysupgrade.conf ${tmpdir}/etc/
copy_if_exists /etc/sysupgrade.d/ ${tmpdir}/etc/
copy_if_exists /etc/.squash_layer_version ${tmpdir}/etc/
echo ${SCRIPT_VERSION} > ${tmpdir}/script_version
if ${EXEC_COMMANDS} ; then
execute_commands
fi
# Finish by system logs to prevent space errors
echo "Extracting system logs"
if [ "${option}" = "minimal" ]; then
copy_if_exists /var/log/messages ${tmpdir}/var/log/
copy_if_exists /var/log/dmesg ${tmpdir}/var/log/
for filename in /var/log/*.1; do
copy_if_exists ${filename} ${tmpdir}/var/log/
done
for filename in /var/log/*.log; do
copy_if_exists ${filename} ${tmpdir}/var/log/
done
else
copy_if_exists /var/log/ ${tmpdir}/var/
fi
echo "Saving all data in ${output_dir}/${name}.tar.gz"
tar -C /user -czf ${output_dir}/${name}.tar.gz ${name}
echo "Done"
sync
rm -rf $tmpdir
}
#####################
# MAIN
#####################
# Detect the environment of execution of the script
detect_env
# Manage scripts arguments
while [[ $# -gt 0 ]]
do
key="$1"
case $key in
-l|--ignore-leds)
LEDS_MANAGEMENT=false
shift
;;
-o|--output-dir)
OUTPUT_DIR="$2"
shift 2
;;
-c|--disable-commands)
EXEC_COMMANDS=false
shift
;;
-h|--help)
show_help
shift
exit 1
;;
-v|--version)
show_version
shift
exit 1
;;
-m|--minimal)
OPTION="minimal"
shift
;;
*)
shift
;;
esac
done
# Check if firmware version is compatible
check_fw_version
# Detect type of platform
detect_platform
# Fill SERIAL and SHORT_MAC variables
fill_serials
# Set led On
if ${LEDS_MANAGEMENT} ; then
detect_leds
set_led none 1
fi
# Get logs
get_logs ${OUTPUT_DIR} ${OPTION}
# End of script, manage user's notifications
if ${LEDS_MANAGEMENT} ; then
# Prepare reboot after USB unplug
cat > /tmp/usbkey_remove << EOF
# Switch off Led
echo "none" > ${LED_PATH}/trigger
EOF
# All is right, make Led blinking fast to notify User
set_led timer 100 100
fi
++++
===== Helium =====
This section contains some tips to do the first level of analysis (support level 1) on Helium gateways.\\
It will help you to investigate before escalate to Kerlink support (level 2).
**Please note that there is an existing open wiki for Helium users: [[https://wikikerlink.fr/helium|Kerlink Helium Hotspot Wiki]]**\\
* First step is to follow [[https://wikikerlink.fr/helium/doku.php?id=home#troubleshooting_guide|Troubleshooting guide]] chapter which contains:
* [[https://wikikerlink.fr/helium/doku.php?id=home#self_diagnostic|Self diagnostic]] section
* [[https://wikikerlink.fr/helium/doku.php?id=home#useful_links|Useful links]] section
* Second step, if necessary, is to follow the table below:
^ Source of info ^ KPI ^ Value ^ Action ^
^ [[https://helium-onboarding.kerlink.com/|Onboarding dashboard]] | Last seen | More than 30m ago | Gateway is offline. Ask customer to check power/internet. |
| ::: | ::: | Less than 30m ago | Gateway is online. Check next KPI. |
| ::: | Block height | = 0 | Gateway is downloading a snapshot. |
| ::: | ::: | = 1 | Gateway is loading the downloaded snapshot. |
| ::: | ::: | Else | KPI is ok, check next KPI. |
| ::: | Block age | > 30 mn | Gateway is syncing, check again in one hour. If it persists, you can open a ticket to Kerlink Support. |
| ::: | ::: | < 30 mn | Gateway is synced. Check next KPI. |
^ [[https://explorer.helium.com//|Helium explorer]] | 24h/14d earnings | > 0 HNT | Seems OK. Confirm with other Explorer KPIs. |
| ::: | Last activity date | < 3h | Seems OK. Confirm with other Explorer KPIs. |
| ::: | Is there a constructed challenge in the past 24h? | Yes | Seems OK. Confirm with other Explorer KPIs. |
| ::: | ::: | No | When gateway is synced, it should challenge. You can open a ticket to Kerlink Support. |
| ::: | Is there a challenged beaconer in the past 24h? | Yes | Seems OK. Confirm with other Explorer KPIs. |
| ::: | ::: | No | Gateway might be relayed. If not, you can open a ticket to Kerlink Support. |
| ::: | Is there a witnessed beacon in the past 8-24h? | Value depends on density around miner | Seems OK. Confirm with other Explorer KPIs. |
| ::: | ::: | No | You can open a ticket to Kerlink Support. |
| ::: | Is there a “broadcasted beacon” in the past 5d? | Yes | Seems OK. Confirm with other Explorer KPIs. |
| ::: | ::: | No | This is random but it seems odd. You can open a ticket to Kerlink Support. |
==== Gateways connected to a WMC====
For gateways connected to a WMC, you can use the Remote Command: Remote Access > Command\\
{{:images:remote_command_wanesy_management_center.png?800|}}
** Due its cryptographic key and to ensure Kerlink Helium Hotspot security, no SSH access can be given to those gateways. \\ **
You can use commands below:
^ Command ^ Awaited result ^
| opkg list | {{:images:opkg.png?400|}} |
| monit status miner | {{:images:monit_status_miner.png?400|}} |Status = OK |
| monit status lorafwd_helium | {{:images:monit_status_lorafwd_helium.png?400|}} | Status = OK |
| /etc/init.d/miner status | {{:images:miner_status.png?400|}} | Check Diff |
If the result is different than the awaited result, you can open a ticket to Kerlink Support.