Degraded network performance after recent update

[iandk]

Describe the bug
I'm using a Pi 4 as a basic Samba share for several months now.
Recently after performing a system update & upgrade I noticed a degraded read and write performance.

It must be related to a recent update since otherwise, nothing has changed.
The write performance has dropped from a constant 113MB/s to around 75MB/s.

Since nothing else has changed in my setup I flashed an older build of Raspberry Pi OS ( 64bit was the only image I could find) to the Pi.
With this version, it's performing very well, just like the last few months.

But after running a system update && upgrade performance is significantly worse again.
The Samba version is exactly the same in both situations.

I guess it's related to the newer kernel and maybe some changes to the underlying network stack?

To reproduce
Configure a basic samba share on some external storage with an older version of Raspberry Pi OS.
I've used this build

After running a system update && upgrade the write performance is much lower than before.

Expected behaviour
More or less stable transfer at 100MB/s+

Actual behaviour
After a few seconds, the write speed to the Pi drops to around 75MB/s and stays there.

System

Raspberry Pi 4 2GB with Crucial MX500 attached via UASP USB3 adapter.
The disk is formatted as XFS and has trim support enabled.

WORKING

Linux raspberrypi 5.4.51-v8+ #1333 SMP PREEMPT Mon Aug 10 16:58:35 BST 2020 aarch64 GNU/Linux

Raspberry Pi reference 2020-08-20
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 7252c154838ec5b4576f29c996ac8fe3750cae12, stage4

Aug 19 2020 17:38:16
Copyright (c) 2012 Broadcom
version e90cba19a98a0d1f2ef086b9cafcbca00778f094 (clean) (release) (start)

NOT WORKING

Linux raspberrypi 5.10.11-v7l+ #1399 SMP Thu Jan 28 12:09:48 GMT 2021 armv7l GNU/Linux

Raspberry Pi reference 2021-01-11
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 21090519d85bdaa1615d5d5057d37b09368ea5d2, stage2

Jan 27 2021 22:19:57
Copyright (c) 2012 Broadcom
version 99d9a48302e4553cff3688692bb7e9ac760a03fa (clean) (release) (start)

Logs
No relevant output in dmesg or syslog

Additional context
I've tested it with three different drives, and two different Pis.
I also reinstalled Raspberry Pi OS several times, with nothing other than samba installed.
Therefore I'm 100% sure it's related to one of the updates.

The disk itself is performing well in both cases.

root@raspberrypi/home # dd if=/dev/zero of=test.data bs=4M count=1K
1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 13.0589 s, 329 MB/s

[mi-hol]

Could you please update your issue with details for the used UASP USB3 adapter from dmesg?
Background:
certain SATA2USB chipset & firmware versions seem to have bugs that impact performance negatively.

example for detail I'm looking for (note: this device is not affected!)
[ 3.013976] usb 2-2: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[ 3.045420] usb 2-2: New USB device found, idVendor=174c, idProduct=55aa, bcdDevice= 1.00
[ 3.053773] usb 2-2: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[ 3.061033] usb 2-2: Product: IB-RD3662-C31
[ 3.065324] usb 2-2: Manufacturer: ICY BOX
[ 3.069496] usb 2-2: SerialNumber: 123456789100

and output of lsusb
Bus 002 Device 002: ID 174c:55aa ASMedia Technology Inc. Name: ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge

[iandk]

I've attached the output below.
At least one guy is experiencing the same issue.

Bus 002 Device 002: ID 174c:55aa ASMedia Technology Inc. Name: ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

[    1.793617] usb 1-1: new high-speed USB device number 2 using xhci_hcd
[    1.976374] usb 1-1: New USB device found, idVendor=2109, idProduct=3431, bcdDevice= 4.21
[    1.976424] usb 1-1: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[    1.976458] usb 1-1: Product: USB2.0 Hub
[    1.979228] hub 1-1:1.0: USB hub found
[    1.979571] hub 1-1:1.0: 4 ports detected
[    2.124045] usb 2-1: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[    2.137014] systemd[1]: System time before build time, advancing clock.
[    2.155349] usb 2-1: New USB device found, idVendor=174c, idProduct=55aa, bcdDevice= 1.00
[    2.155393] usb 2-1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[    2.155427] usb 2-1: Product: Ugreen Storage Device
[    2.155458] usb 2-1: Manufacturer: Ugreen
[    2.155487] usb 2-1: SerialNumber: 091912070139
[    2.182292] scsi host0: uas
[    2.184443] scsi 0:0:0:0: Direct-Access     CT250MX5 00SSD1           0    PQ: 0 ANSI: 6
[    2.187262] sd 0:0:0:0: [sda] 488397168 512-byte logical blocks: (250 GB/233 GiB)
[    2.187500] sd 0:0:0:0: [sda] Write Protect is off
[    2.187537] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
[    2.187922] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.188686] sd 0:0:0:0: [sda] Optimal transfer size 33553920 bytes
[    2.235616]  sda: sda1
[    2.238716] sd 0:0:0:0: [sda] Attached SCSI disk

[mi-hol]

Update 10Feb2021 16:49 UTC
I've collected performance metrics using CrystalDiskMark previously and can NOT confirm a significant drop after upgrade to kernel 5.10.11 for my openmediavault based NAS
(i.e. read of 64MB file increased from 93 MB/s to 109 MB/s) on the same hardware & setup (details (incl SAMBA config) here)

[iandk]

64MB file?
You're not testing the throughput to your disk, you're testing your cache.

Try again with a larger file and several read/ write operations.

[iandk]

I just tested it again.

Flashed the newest Raspberry Pi OS 32bit version (still running the old 5.4.83-v7l+ Kernel ), configured a basic samba share and ran a few tests with around 10-20GB of data.
No issues at all, the transfer speed was at 110MB/s all the time.

After that, I updated to the kernel version 5.10.11-v7l+ and updated the preinstalled packages as well.
I then ran exactly the same test again, nothing else changed.
And the transfer speed dropped after around 3-4GB from 110MB/s to 75MB/s.

This confirms that it's not related to a config issue or some other factor.
I ran exactly the same test again, the only thing that has changed is the kernel version.

[popcornmix]

Can you test if the issue is samba related or network related?
E.g. measure network speed with iperf between the two machines?

[iandk]

It doesn't seem to be network related.
iPerf shows a constant 950mbit/s in both directions.

Edit: Samba itself shouldn't be the issue either, since the version hasn't changed with the update.

[s3b-s]

I am having a similar issue with my NFS shares that I am using for my kodi box. Everything was working just fine, but at the moment the video playback is impossible. My rpi4 is currently running: Linux 5.10.14-v7l+ #1401 SMP Mon Feb 8 14:27:07 GMT 2021 armv7l

Before the update everything was working fine.
I even turned on my old rpi3 board with :5.4.79-v7+ #1373 SMP Mon Nov 23 13:22:33 GMT 2020 armv7l and just copied over my /etc/exports. Suddenly kodi went back to life.

Strangely enough, when accessing the share from my windows laptop, on both the video playback is smooth as butter.

This is strange as for example iperf3 shows stable 900 Mb/s+.
For a test I have played the same movie from both machines, while it played just fine from the RPI3 board, the RPI4 is just full of stutter and after a while it crushes.

RPI4 nfsstat after 2min:

Server rpc stats:
calls      badcalls   badfmt     badauth    badclnt
61554      0          0          0          0

Server nfs v3:
null             getattr          setattr          lookup           access
7         0%     436       0%     0         0%     28117    45%     194       0%
readlink         read             write            create           mkdir
14        0%     32737    53%     0         0%     0         0%     0         0%
symlink          mknod            remove           rmdir            rename
0         0%     0         0%     0         0%     0         0%     0         0%
link             readdir          readdirplus      fsstat           fsinfo
0         0%     0         0%     39        0%     1         0%     8         0%
pathconf         commit

RPI3 after 2 min playback:

Server rpc stats:
calls      badcalls   badfmt     badauth    badclnt
1698       0          0          0          0

Server nfs v3:
null             getattr          setattr          lookup           access
1         0%     5         0%     0         0%     97        5%     3         0%
readlink         read             write            create           mkdir
0         0%     1589     93%     0         0%     0         0%     0         0%
symlink          mknod            remove           rmdir            rename
0         0%     0         0%     0         0%     0         0%     0         0%
link             readdir          readdirplus      fsstat           fsinfo
0         0%     0         0%     2         0%     0         0%     1         0%
pathconf         commit
0         0%     0         0%

[popcornmix]

@dq-git your issue may be xbmc/xbmc#19147

[d33pd1ve]

I have the same problem with my Pi 4 4GB!

I use it as a NAS with an 8TB WD Elements attached.
Performance is much lower after the recent updates

[iandk]

This is also the second time I'm noticing IO errors after the kernel update.
I've never seen this issue in the past 6 months with the old kernel.

[59761.546841] sd 1:0:0:0: [sdb] Attached SCSI disk
[152744.813849] XFS (sda1): log I/O error -5
[152744.813945] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1196 of file fs/xfs/xfs_log.c. Return address = 78401cd3
[152744.813963] XFS (sda1): Log I/O Error Detected. Shutting down filesystem
[152744.813996] XFS (sda1): Please unmount the filesystem and rectify the problem(s)

[s3b-s]

@popcornmix Thanks, it looks like this is just the case here. Meanwhile I reverted kernel to the same that was working on rpi3 and magic happened, everything works now.

[jorensanbar]

Same bug.

Upgrated to kernel 5.10.11 speed: 70 MB/s.
​
Returning to 5.4.79-v71+ #1373, speed: 115 MB/s

Pi 4B 2GB + WD Elements 5TB USB 3.0

[jorensanbar]

Upgrated to kernel 5.10.11 speed: 70 MB/s.

New test: It is not the kernel, it is another update.

Continuous speed problem, after updating everything except the Kernel.

My OMV, is working with an old kernel, and after updating everything else, the problem starts.

[iandk]

Oh.. I was quite sure it’s related to the kernel.
I’ll have a look at the packages that will get updated