Advanced¶

The following section covers advanced troubleshooting techniques. flexiWAN relies on number of underlying components, from Ubuntu Server as OS and all its functionalites such as netplan, to VPP and FRR components.

VPP troubleshooting¶

Packet capture¶

flexiWAN utilizes VPP, so packet capture using tcpdump will not show all traffic. Instead use the following commands to capture packets. This can also be done from flexiManage Send Command tab.

Baremetal:

vppctl pcap dispatch trace on max 1000 file vrouter.pcap buffer-trace dpdk-input 1000
wait 10-15 seconds
vppctl pcap dispatch trace off

VMware:

vppctl pcap dispatch trace on max 10000 file vrouter.pcap buffer-trace vmxnet3-input 1000
wait 10-15 seconds
vppctl pcap dispatch trace off

After executing the above commands, download the .pcap file from /tmp. In case the file doesn’t appear, simply re-run the packet capture commands again. Inspect the pcap using Wireshark. With the VPP pcap each packet appears multiple times showing the path of the packet through the VPP nodes. This is useful to troubleshoot network issues and tunnel connectivity issues.

VPP CLI¶

flexiWAN uses VPP network stack. VPP CLI and vppctl commands are available from flexiEdge shell. Learn more about VPP CLI and full list of supported commands here.

Most commonly used vppctl commnands:

vppctl show int

vppctl show hard

vppctl show ip addr

vppctl show ip fib

vppctl arp

vppctl show ip arp

OSPF troubleshooting¶

flexiWAN offers FRR for routing, and its component OSPF to share routing information between sites. The way the system works is that OSPF learns possible shortest paths routes. In this section we cover OSPF troubleshooting basics. Commands can be executed from shell or from the Send Command tab per device.

Check if ospf process is running:

ps -ef | grep ospf

Capture OSPF packets using tcpdump on flexiEdge:

tcpdump -n -v -s0 -c 10 -i <Linux i/f>:nnnp proto ospf - capture 10 ospf packets

To troubleshoot FRR use the vtysh:

Show current FRR configuration:

vtysh -c "show running-config"

Show learned neighbours:

vtysh -c "show ip ospf neighbor"

Show interfaces:

vtysh -c "show ip ospf interface"

Show routes:

vtysh -c "show ip ospf route"

Warning

Manually editing OSPF is not supported. Please make all changes through flexiManage instead.

BGP troubleshooting¶

flexiWAN supports BGP for routing. Several status commands are supported via vtysh. The following commands can be executed via Command tab from flexiMaange.

Summary of BGP neighbor status:

vtysh -c "show bgp summary"

BGP nexthop table:

vtysh -c "show bgp nexthop"

Display number of prefixes for all afi/safi:

vtysh -c "statistics-all"

Show BGP VRFs:

vtysh -c "vrfs"

Global BGP memory statistics:

vtysh -c "show bgp memory"

Netplan configuration¶

flexiWAN uses netplan.io for network interfaces configuration. Through Netplan YAML files each interface can be configured. Learn more about Netplan here.

During Ubuntu installation user is prompted to select a network interface for internet access. The interface which is selected during setup will be automatically defined in the default Netplan YAML file. This file is used when the vRouter is not running.

/etc/netplan/50-cloud-init.yaml

Interfaces configuration through flexiManage is saved in netplan files once the vRouter is started. flexiManage does not change unassigned interfaces.

UDP Connection Troubleshooting¶

Checking tunnel network connectivity with UDP test

In case of packet captures showing flexiWAN tunnel traffic is not arriving, run the following test to see if UDP port 4789 is open and not filtered. The test consist of running UDP server script on site A and sending UDP traffic to it from site B. Use the public IP and ports in the case of NAT as seen in the flexiManage public IP on the Device -> Interfaces page.

The test in question uses phython3 so make sure to first install pip3:

apt install python3-pip

Then follow the steps below on each of the flexiEdge devices.

Install the tool

pip3 install udp-test

stop the vRouter from flexiManage devices page on both routers in question.
On site A start the server:

udp-test server -p 4789

Connect to server on site A from site B

udp-test client -h server_IP -p 4789 -l 4789

Where server_IP is IP of the remote site. Try sending messages and confirm it’s passing.

Path Selection Troubleshooting¶

This section will explain how to check the policies in Flexiwan Edge/Router using basic VPP commands

Once you configure the policies on the Path Selection page you need to install into the respective Flexiwan Edge/Router, Once you install the policies FlexiManage will push the configuration to the respective Flexiwan Edge/Router.

To check whether the policies are pushed we can use the following VPP CLI commands

VPP CLI Commands to check Policy

To check if LAN outgoing traffic is matching the policy, run the following command:

vppctl show fwabf policy

The output should be:

The above command shows few important counters:

matched - Number of packets matched the policy defined
applied - Number of packets for which the policy is applied
fallback - Number of packets for which the policy is not matched and following fallback action
dropped - Number of packets dropped by applying the policy.
acl: - Displays acl index number, note it for for the next command.

vppctl show acl-plugin acl index <acl index number> - replace acl index number from the output of above command.

The above command will show the access control list of a specific ACL index number.

Firewall Troubleshooting¶

flexiWAN offers several commands to display information about firewall and NAT.

vppctl show nat44 sessions - view current WAN NAT sessions
vppctl show acl-plugin interface - list interfaces with ACL index values. To be used with the next command.
vppctl show acl-plugin acl <index> - view allowed / blocked per ACL rule. Under “index” put the interface from the above command.

IPSec / IKEv2 Troubleshooting¶

To view status of IKEv2 connections enter the following command from the device Command tab or using shell:

vppctl show ikev2 sa details

Advanced logging may be set running the following commands via Command tab or shell:

vppctl ikev2 set logging level 5
vppctl event-logger clear
vppctl show event-logger

After entering the above commands, IKEv2/IPsec logging will be outputed to the device syslog. Syslog can be fetched from flexiManage, by navigating to device Logs tab.

PPPoE DNS settings¶

To view the DNS servers used by PPPoE conection enter the following command via shell only.

systemd-resolve --status

The command will output DNS IP’s set for all interfaces, search for the one with PPP in name, for example “ppp-eth0”.

LTE Troubleshooting¶

flexiWAN includes several commands to troubleshoot and manipulate LTE settings from the command line. First find out the device names by entering the following command:

ls -l /dev/cdc-*

Output will show device name /dev/cdc-wdm0 or in case of multiple modems a list where each will follow the numbering cdc-wdm0, cdc-wdm1 etc.

Get mobile provider name:

mbimcli --device /dev/cdc-wdm0 -p --query-subscriber-ready-status
mbimcli --device /dev/cdc-wdm0 -p --query-registration-state

Query IP address:

mbimcli --device /dev/cdc-wdm0 -p --query-ip-configuration

Show SIM card status:

qmicli --device=/dev/cdc-wdm0 -p --uim-get-card-status

Output for “Card state” should be present.

Switching SIM slots or when changing SIM card in any slot, run the following commands:

qmicli --device=/dev/cdc-wdm0 -p --uim-sim-power-off=1
qmicli --device=/dev/cdc-wdm0 -p --uim-sim-power-on=1

Disconnect from provider network:

mbimcli --device /dev/cdc-wdm0 -p --disconnect=0

Connect to the provider network:

mbimcli --device /dev/cdc-wdm0 -p --query-subscriber-ready-status
mbimcli --device /dev/cdc-wdm0 -p --query-registration-state
mbimcli --device /dev/cdc-wdm0 -p --attach-packet-service
mbimcli --device /dev/cdc-wdm0 -p --connect=apn=APNNAME,username=USERNAME,password=PASS,auth=PAP

For step 4. replace APNNAME, USERNAME, PASS with provider info. Supported auth methods are: PAP, CHAP and MSCHAPV2.

Note, use the commands above only for troubleshooting, in case when the device shows LTE as not connected. In most cases, LTE should automatically connect.

List all other available:

qmicli --help-all

Note, not all other commands may work or are supported.

Quectel specific steps¶

Certain Quectel modules such as Quectel RM500Q-AE may experience issue when SIM card isn’t detected. A specific modem configuration is required before registering the device to flexiManage. Note, if the device is already registered, simply delete it it from flexiManage and re-register after completing the steps below.

First step is to install minicom to access the modem using its serial port.

apt install minicom

After installtion run minicom -s to start minicom. Then select Serial port setup.

On the next screen type A on the keyboard to select Serial Device and change it to /dev/ttyUSB2. Press enter twice to confirm.

Finally, select Save setup as dfl and select Exit from Minicom. To verify sim card can now be detected, enter minicom -s again and run the following commands:

AT - to make sure modem is responding to commands.
AT+QUIMSLOT? - the expected output is 1. If the value is 2. follow the steps 3. and 4.
AT+QUIMSLOT=1
AT+QPOWD=0 - the following command restarts the modem. After modem restart re-try step 2.

QMI and MBIM switching¶

flexiWAN can switch supported LTE modems from QMI to MBIM and vice versa from the command line.

Please add to our docs how to switch modem from QMI to MBIM, and from MBIM to QMI.

QMI to MBIM¶

Sierra Wireless

qmicli --device=/dev/cdc-wdm0 --device-open-proxy --dms-swi-set-usb-composition=8
qmicli --device=/dev/cdc-wdm0 --device-open-proxy --dms-set-operating-mode=offline
qmicli --device=/dev/cdc-wdm0 --device-open-proxy --dms-set-operating-mode=reset

Quectel

Use minicom, run:

AT+QCFG="usbnet",2
AT+QPOWD=0

MBIM to QMI¶

Sierra Wireless

qmicli --device=/dev/cdc-wdm0 --device-open-proxy --dms-swi-set-usb-composition=6
qmicli --device=/dev/cdc-wdm0 --device-open-proxy --dms-set-operating-mode=offline
qmicli --device=/dev/cdc-wdm0 --device-open-proxy --dms-set-operating-mode=reset

Quectel

Use minicom, run:

AT+QCFG="usbnet",0
AT+QPOWD=0

Resolving Host OS upgrade issues¶

flexiWAN 6.2.1 introduced new Host OS which upgrade should be painless and fast. However, in some rare cases device may not be bootable due kernel issue. If an error in boot is as following, please follow the steps bellow to resolution for a quick fix.

end kernel panic not syncing vfs unable to mount root fs on unknown blocked

Reboot again the device
On the bootloader (GRUB) menu, select the previous boot entry, the one with second latest Linux kernel version available
Once the device has been booted up properly, login to the device with admin user and run the following commands to recover from that situation:

sudo update-initramfs -u -k all
sudo update-grub

Grab the output of the above commands.
Reboot again the device by executing reboot.

flexiWAN should boot now normally.