Skip to content

net: traffic blocks when running ingress UDP performance tests with very large buffers #1754

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
raduiliescu opened this issue Apr 7, 2020 · 1 comment
Assignees
Labels
Priority: Medium Indicates than an issue or pull request should be resolved ahead of issues or pull requests labelled Type: Bug Indicates an unexpected problem or unintended behavior

Comments

@raduiliescu
Copy link
Contributor

raduiliescu commented Apr 7, 2020

On a m5.metal host running Amazon Linux, create a microVM with 8 cores and one network interface using xenial rootfs
Use the recommended network setup for host and guest - https://github.com/firecracker-microvm/firecracker/blob/master/docs/network-setup.md

Increase RX buffers in VM by 100 times
echo 21299200 > /proc/sys/net/core/rmem_max
echo 21299200 > /proc/sys/net/core/rmem_default

Start an iperf3 server on VM - iperf3 -p 11111 -s 172.16.0.2 -D
Start iperf3 udp client on host: iperf3 -p 11111 -u -c 172.16.0.2 -V -b 1000000000 -t 10000

After some time all networking of the VM will stop working. This happens with latest code as well as with latest version.

[ 4] 36.00-37.00 sec 119 MBytes 1.00 Gbits/sec 86332
[ 4] 37.00-38.00 sec 119 MBytes 999 Mbits/sec 86263
[ 4] 38.00-39.00 sec 119 MBytes 1.00 Gbits/sec 86326
[ 4] 39.00-40.00 sec 59.7 MBytes 501 Mbits/sec 43216
[ 4] 40.00-41.00 sec 0.00 Bytes 0.00 bits/sec 0
[ 4] 41.00-42.00 sec 0.00 Bytes 0.00 bits/sec 0
[ 4] 42.00-43.00 sec 66.5 KBytes 544 Kbits/sec 47
[ 4] 43.00-44.00 sec 0.00 Bytes 0.00 bits/sec 0

Script to start the VM:

#!/bin/sh

arch=`uname -m`
kernel_path=$(pwd)"/hello-vmlinux.bin"

if [ ${arch} = "x86_64" ]; then
	curl --unix-socket /tmp/firecracker.socket -i \
        -X PUT 'http://localhost/boot-source'   \
        -H 'Accept: application/json'           \
        -H 'Content-Type: application/json'     \
        -d "{
              \"kernel_image_path\": \"${kernel_path}\",
              \"boot_args\": \"console=ttyS0 reboot=k panic=1 pci=off\"
         }"
elif [ ${arch} = "aarch64" ]; then
    curl --unix-socket /tmp/firecracker.socket -i \
        -X PUT 'http://localhost/boot-source'   \
        -H 'Accept: application/json'           \
        -H 'Content-Type: application/json'     \
        -d "{
              \"kernel_image_path\": \"${kernel_path}\",
              \"boot_args\": \"keep_bootcon console=ttyS0 reboot=k panic=1 pci=off\"
         }"
else
    echo "Cannot run firecracker on $arch architecture!"
    exit 1
fi

rootfs_path=$(pwd)"/xenial.rootfs.ext4"
#rootfs_path=$(pwd)"/hello-rootfs.ext4"
curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT 'http://localhost/drives/rootfs' \
    -H 'Accept: application/json'           \
    -H 'Content-Type: application/json'     \
    -d "{
          \"drive_id\": \"rootfs\",
          \"path_on_host\": \"${rootfs_path}\",
          \"is_root_device\": true,
          \"is_read_only\": false
     }"

curl --unix-socket /tmp/firecracker.socket -i  \
    -X PUT 'http://localhost/machine-config' \
    -H 'Accept: application/json'            \
    -H 'Content-Type: application/json'      \
    -d '{
        "vcpu_count": 8,
        "mem_size_mib": 4096,
        "ht_enabled": false
    }'

curl --unix-socket /tmp/firecracker.socket -i \
     -X PUT 'http://localhost/network-interfaces/eth0' \
     -H 'Accept: application/json' \
     -H 'Content-Type: application/json' \
       -d '{
      "iface_id": "eth0",
      "guest_mac": "AA:FC:00:00:00:01",
      "host_dev_name": "tap0"
    }'

curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT 'http://localhost/actions'       \
    -H  'Accept: application/json'          \
    -H  'Content-Type: application/json'    \
    -d '{
        "action_type": "InstanceStart"
     }'
@dianpopa dianpopa added the Type: Bug Indicates an unexpected problem or unintended behavior label Apr 7, 2020
@sandreim sandreim added the Priority: Medium Indicates than an issue or pull request should be resolved ahead of issues or pull requests labelled label Apr 8, 2020
@serban300 serban300 self-assigned this May 14, 2020
@ioanachirca
Copy link
Contributor

Duplicate of #2057

@ioanachirca ioanachirca marked this as a duplicate of #2057 Aug 3, 2020
ioanachirca pushed a commit to ioanachirca/firecracker that referenced this issue Aug 3, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
ioanachirca pushed a commit to ioanachirca/firecracker that referenced this issue Aug 3, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
ioanachirca pushed a commit to ioanachirca/firecracker that referenced this issue Aug 3, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
dianpopa pushed a commit to dianpopa/firecracker that referenced this issue Aug 3, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
ioanachirca pushed a commit to ioanachirca/firecracker that referenced this issue Aug 3, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
dianpopa pushed a commit that referenced this issue Aug 4, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
dianpopa pushed a commit that referenced this issue Aug 4, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
dianpopa pushed a commit to dianpopa/firecracker that referenced this issue Aug 4, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
dianpopa pushed a commit to dianpopa/firecracker that referenced this issue Aug 4, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
dianpopa pushed a commit to dianpopa/firecracker that referenced this issue Aug 4, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
ioanachirca pushed a commit to ioanachirca/firecracker that referenced this issue Aug 4, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
ioanachirca pushed a commit to ioanachirca/firecracker that referenced this issue Aug 4, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
dianpopa pushed a commit to dianpopa/firecracker that referenced this issue Aug 4, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
dianpopa pushed a commit to dianpopa/firecracker that referenced this issue Aug 4, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
ioanachirca pushed a commit that referenced this issue Aug 4, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
dianpopa pushed a commit to dianpopa/firecracker that referenced this issue Aug 5, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
dianpopa pushed a commit to dianpopa/firecracker that referenced this issue Aug 5, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
dianpopa pushed a commit to dianpopa/firecracker that referenced this issue Aug 5, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
dianpopa pushed a commit that referenced this issue Aug 5, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
dianpopa pushed a commit to dianpopa/firecracker that referenced this issue Dec 23, 2020
Validate that the RX queue won't freeze under high stateless ingress
traffic when encountering bottlenecks on the tap.

Signed-off-by: Serban Iorga <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Priority: Medium Indicates than an issue or pull request should be resolved ahead of issues or pull requests labelled Type: Bug Indicates an unexpected problem or unintended behavior
Projects
None yet
Development

No branches or pull requests

5 participants