Added guidelines for scaling firecracker VMs count

Author: ustiugov

docs/scaling.md

@@ -0,0 +1,50 @@
+# Scaling the number of Firecracker microVMs per host
+
+To scale the number of microVMs past one thousand, one needs to properly configure 
+the system contraints and provision enough networking resources for the microVMs.
+
+On Ubuntu 18.04 Linux, one needs to the number of processes, threads, memory, 
+open files that may simultaneously exist in a system as well as create enough 
+virtual bridges (one bridge can serve up to 1023 interfaces).
+
+To configure the system, one needs to set up the following parameters. 
+Note that exact values depend on your setting.
+
+In `/etc/security/limits.conf`, set `nofile`, `nproc` and `stack` to the 
+appropriate values for both normal users and root:
+```
+sudo sh -c "echo \"* soft nofile 1000000\" >> /etc/security/limits.conf"
+sudo sh -c "echo \"* hard nofile 1000000\" >> /etc/security/limits.conf"
+sudo sh -c "echo \"root soft nofile 1000000\" >> /etc/security/limits.conf"
+sudo sh -c "echo \"root hard nofile 1000000\" >> /etc/security/limits.conf"
+sudo sh -c "echo \"* soft nproc 4000000\" >> /etc/security/limits.conf"
+sudo sh -c "echo \"* hard nproc 4000000\" >> /etc/security/limits.conf"
+sudo sh -c "echo \"root soft nproc 4000000\" >> /etc/security/limits.conf"
+sudo sh -c "echo \"root hard nproc 4000000\" >> /etc/security/limits.conf"
+sudo sh -c "echo \"* soft stack 65536\" >> /etc/security/limits.conf"
+sudo sh -c "echo \"* hard stack 65536\" >> /etc/security/limits.conf"
+sudo sh -c "echo \"root soft stack 65536\" >> /etc/security/limits.conf"
+sudo sh -c "echo \"root hard stack 65536\" >> /etc/security/limits.conf"
+```
+
+Additionally, one needs to provision the ARP cache to avoid garbage collection.
+```
+sudo sysctl -w net.ipv4.neigh.default.gc_thresh1=1024
+sudo sysctl -w net.ipv4.neigh.default.gc_thresh2=2048
+sudo sysctl -w net.ipv4.neigh.default.gc_thresh3=4096
+sudo sysctl -w net.ipv4.ip_local_port_range="32769 65535"
+```
+
+Also, configure the maximum number of processes and threads in the system.
+```
+sudo sysctl -w kernel.pid_max=4194303
+sudo sysctl -w kernel.threads-max=999999999
+```
+
+Finally, configure the number of tasks. 
+To configure system-wide, uncomment and set `DefaultTasksMax=infinity`in `/etc/systemd/system.conf`.
+One also needs to set `UsersTasksMax=4000000000` in `/etc/systemd/logind.conf` 
+(note that `infinity` is not a valid value here).
+
+To configure the bridges for CNI, take a look at `demo-network` target in [Makefile](https://github.com/firecracker-microvm/firecracker-containerd/blob/master/Makefile)
+and replicate the code to create enough bridges (1 bridge can have up to 1023 interfaces attached).