VLX is lagging: how to benchmark ReadWrite performance of external SSD

Introduction

Problem description

When scanning with the NavVis VLX all data is directly stored on the external SSD. Persistent lagging issues can therefore be caused by abnormal write/read speeds on this SSD. This instruction explains how to test if the SSD might be causing the lagging.

The route cause of the NaVis VLX lagging can be caused by many different issues. These can be issues with the RAM, NUC, sensor connections and software processes. Use this article to exclude any SSD performance issues when the root cause is unclear and a hardware replacement is considered the next step.

Scope

Applies to: Mobile Mapping Systems incl. VLX

Troubleshooting procedure

1 Log into the device remotely incl. screen output

Do this via teamviewer or ssh.

2 Install stress package on the device

Open terminal #1 on the device and run

sudo apt-get install stress

This will install a package, which puts high load on a specific number of CPU kernels. During VLX mappings all kernels are under heavy load as well.

3 Copy VLX_testbench to the device

Get our internal D&E tool "VLX_testbench" (VLX - testbench) and copy it to the device. You can download it directly (within NavVis VPN) from here: KBA Review Process
From it the CAM_BURST test will be used to put read/write stress to the internal SSD's temp folder.

Here you find how to make the tool executable VLX Tech support tools (DevTools) and usage

4 Open the Ubuntu Disk Utility

1. Go to Disks:

2. Select the external SSD (usually the second one) and choose to benchmark the biggest partition on the disk:

3. Click on "Start Benchmark" to be able to choose settings:

4. Choose the following settings (without starting the benchmark yet):

5 Run the test

Do the following steps right one after another, so that they all run during the SSD benchmarking test:

1. Open a new terminal

2. In terminal #0 run the command htop to monitor CPU load and RAM storage. 

3. In terminal #1 run

   stress -c 6

4. Finally click on "Start Benchmark" within the disk UI.

5. Start a CAM_BURST test in VLX_testbench

   • Open a new terminal #2 and execute the VLX testbench including it's CAM_BURST functionality writing images to the local tmp folder. Execute the following command:

     ./vlx_testbench --no-interaction --skip-test=INTERACTION,BIOS,BLUETOOTH,WIFI,VELODYNE,IMU,CAM_CABLES,BELTHOOK --verbose --cycle 1

6. Take screenshots of the test results and copy all logs

   Example screenshot of a bad SSD on G9-008. Notice the sudden drop in write speed to around 50% of what it should and can be. Depending on the model of the SSD the maximum read and write speed can vary according to specifications (Google!).

6 Format the SSD and test again OR Get a new SSD

If you notice any strange behavior in read or write speeds, try to format the SSD first, then repeat the test and if that doesn't help, get a new one.

Please also note down or take a screenshot of the SSD serial number from Disk Utility tool to verify in case the test is done multiple times:

Note You can also try to get rid of the lagging by just telling the customer to use a different SSD. However, this test helps anyway to confirm or exclude the problematic SSD behavior. (And customers might tell you they did change SSD, despite they didn't because they're convinced it's not the SSD but poor VLX performance...)