firecracker-microvm · sandreim · Oct 28, 2020 · Oct 19, 2020 · Oct 16, 2020 · Oct 16, 2020
diff --git a/.mailmap b/.mailmap
@@ -11,3 +11,7 @@ Tamio-Vesa Nakajima <[email protected]> <[email protected]>
 Iulian Barbu <[email protected]>
 Petre Eftime <[email protected]> <[email protected]>
 karthik nedunchezhiyan <[email protected]> <[email protected]>
+Alin Dima <[email protected]>
+Andrei Sandu <[email protected]> <[email protected]>
+Diana Popa <[email protected]> <[email protected]>
+Alexandru Cihodaru <[email protected]>
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,15 @@
 
 ### Added
 
+- Added devtool test `-c|--cpuset-cpus` flag for cpus confinement when tests
+  run.
+- Added devtool test `-m|--cpuset-mems` flag for memory confinement when tests
+  run.
+
+## [0.23.0]
+
+### Added
+
 - Added metric for throttled block device events.
 - Added metrics for counting rate limiter throttling events.
 - Added metric for counting MAC address updates.
@@ -30,10 +39,8 @@
 - Added a new API call, `PUT /snapshot/load`, for loading a snapshot.
 - Added new jailer command line argument `--cgroup` which allow the user to
   specify the cgroups that are going to be set by the Jailer.
-- Added devtool test `-c|--cpuset-cpus` flag for cpus confinement when tests
-  run.
-- Added devtool test `-m|--cpuset-mems` flag for memory confinement when tests
-  run.
+- Added full support for AMD CPUs (General Availability). More details
+  [here](README.md#supported-platforms).
 
 ### Fixed
 

diff --git a/CREDITS.md b/CREDITS.md
@@ -21,13 +21,16 @@ Contributors to the Firecracker repository:
 * Aaron Hill <[email protected]>
 * Abhijeet Kasurde <[email protected]>
 * Adrian Catangiu <[email protected]>
+* Ahmed Abouzied <[email protected]>
 * Alakesh <[email protected]>
 * Aleksa Sarai <[email protected]>
 * Alex Chan <[email protected]>
 * Alex Glikson <[email protected]>
 * Alexandra Iordache <[email protected]>
 * Alexandru Agache <[email protected]>
 * Alexandru Branciog <[email protected]>
+* Alexandru Cihodaru <[email protected]>
+* Alin Dima <[email protected]>
 * Andreea Florescu <[email protected]>
 * Andrei Casu-Pop <[email protected]>
 * Andrei Cipu <[email protected]>
@@ -38,21 +41,30 @@ Contributors to the Firecracker repository:
 * Atsushi Ishibashi <[email protected]>
 * Aussie Schnore <[email protected]>
 * Babis Chalios <[email protected]>
+* Begley Brothers Inc <[email protected]>
+* Benjamin Fry <[email protected]>
+* bin liu <[email protected]>
 * Bob Potter <[email protected]>
 * Bogdan Ionita <[email protected]>
+* Caleb Albers <[email protected]>
+* Cam Mannett <[email protected]>
 * chaos matrix <[email protected]>
 * Chinmay Kousik <[email protected]>
 * Chris Christensen <[email protected]>
+* Christian González <[email protected]>
 * Christopher Diehl <[email protected]>
 * cneira <[email protected]>
 * Constantin Musca <[email protected]>
+* Damien Stanton <[email protected]>
 * Dan Horobeanu <[email protected]>
 * Dan Lemmond <[email protected]>
 * Deepesh Pathak <[email protected]>
+* defunct <[email protected]>
 * Denis Andrejew <[email protected]>
 * Diana Popa <[email protected]>
 * Dmitrii <[email protected]>
 * Filippo Sironi <[email protected]>
+* Fraser Pringle <[email protected]>
 * Gabe Jackson <[email protected]>
 * Gabriel Ionescu <[email protected]>
 * Garrett Squire <[email protected]>
@@ -70,7 +82,10 @@ Contributors to the Firecracker repository:
 * Iulian Barbu <[email protected]>
 * James Turnbull <[email protected]>
 * Javier Romero <[email protected]>
+* jonas serrano <[email protected]>
 * Josh Abraham <[email protected]>
+* Josh McConnell <[email protected]>
+* Joshua Abraham <[email protected]>
 * Julian Stecklina <[email protected]>
 * karthik nedunchezhiyan <[email protected]>
 * KarthikVelayutham <[email protected]>
@@ -81,22 +96,31 @@ Contributors to the Firecracker repository:
 * Liu Jiang <[email protected]>
 * Lloyd <[email protected]>
 * lloydmeta <[email protected]>
+* LOU Xun <[email protected]>
+* Luminita Voicu <[email protected]>
 * maciejhirsz <[email protected]>
+* Malhar Vora <[email protected]>
 * Manohar Castelino <[email protected]>
 * Marc Brooker <[email protected]>
 * Marco Vedovati <[email protected]>
 * Masatoshi Higuchi <[email protected]>
 * Massimiliano Torromeo <[email protected]>
 * Matt Wilson <[email protected]>
 * Mehrdad Arshad Rad <[email protected]>
+* Michael Saah <[email protected]>
+* Mihai Stan <[email protected]>
+* moricho <[email protected]>
 * Nathan Hoang <[email protected]>
 * Nathan Sizemore <[email protected]>
 * Nicolas Mesa <[email protected]>
+* Nikolay Edigaryev <[email protected]>
 * Noah Meyerhans <[email protected]>
+* not required <[email protected]>
 * Peng Tao <[email protected]>
 * Penny Zheng <[email protected]>
 * Peter Hrvola <[email protected]>
 * Petre Eftime <[email protected]>
+* Radu Iliescu <[email protected]>
 * Radu Matei Lăcraru <[email protected]>
 * Radu Weiss <[email protected]>
 * Ram Sripracha <[email protected]>
@@ -111,16 +135,21 @@ Contributors to the Firecracker repository:
 * Serban Iorga <[email protected]>
 * shakram02 <[email protected]>
 * Shen Jiale <[email protected]>
+* Shion Yamashita <[email protected]>
 * singwm <[email protected]>
 * Sripracha <[email protected]>
+* Stefan Nita <[email protected]>
 * Tamio-Vesa Nakajima <[email protected]>
 * tidux <[email protected]>
 * Tim Bannister <[email protected]>
 * Tim Deegan <[email protected]>
+* timvisee <[email protected]>
 * Tyler Anton <[email protected]>
 * Urvil Patel <[email protected]>
+* Wei Yang <[email protected]>
 * Weixiao Huang <[email protected]>
 * Wesley Norris <[email protected]>
+* wt-l00 <[email protected]>
 * xibz <[email protected]>
 * xiekeyang <[email protected]>
 * YLyu <[email protected]>

diff --git a/Cargo.lock b/Cargo.lock
diff --git a/FAQ.md b/FAQ.md
@@ -36,10 +36,12 @@ to integrate with container ecosystems.
 
 ### What processors does Firecracker support?
 
-The Firecracker VMM is built to be processor agnostic. Intel processors are
-supported for production workloads. Support for AMD and Arm processors is in
+The Firecracker VMM is built to be processor agnostic. Intel and AMD processors
+are supported for production workloads. Support for Arm processors is in
 developer preview.
 
+You can find more details [here](README.md#supported-platforms).
+
 ### Can Firecracker be used within the container ecosystem?
 
 Yes. Firecracker is integrated with

diff --git a/README.md b/README.md
@@ -14,13 +14,13 @@ in lightweight virtual machines, called microVMs, which combine the security and
 isolation properties provided by hardware virtualization technology with the
 speed and flexibility of containers.
 
+## Overview
 The main component of Firecracker is a virtual machine monitor (VMM) that uses
 the Linux Kernel Virtual Machine (KVM) to create and run microVMs. Firecracker
 has a minimalist design. It excludes unnecessary devices and guest-facing
 functionality to reduce the memory footprint and attack surface area of each
 microVM. This improves security, decreases the startup time, and increases
-hardware utilization. Firecracker currently supports Intel, AMD (preview) and
-Arm (preview) CPUs. Firecracker has also been integrated in container runtimes,
+hardware utilization. Firecracker has also been integrated in container runtimes,
 for example
 [Kata Containers](https://github.com/kata-containers/documentation/wiki/Initial-release-of-Kata-Containers-with-Firecracker-support)
 and [Weaveworks Ignite](https://github.com/weaveworks/ignite).
@@ -120,6 +120,27 @@ The **API endpoint** can be used to:
   scenarios; applies a cgroup/namespace isolation barrier and then
   drops privileges.
 
+## Supported platforms
+
+We continuously test Firecracker on machines with the following CPUs 
+micro-architectures: Intel Skylake, Intel Cascade Lake, AMD Zen2, ARM Cortex-A 
+aarch64.
+
+Firecracker is [generally available](docs/RELEASE_POLICY.md) on Intel x86_64 
+and AMD x86_64 CPUs that offer hardware virtualization support, and that are 
+released starting with 2015. All production use cases should follow [these 
+production host setup instructions](docs/prod-host-setup.md).
+
+Firecracker is in [developer preview](docs/RELEASE_POLICY.md) (and not 
+supported for production workloads) on CPUs based on Arm Cortex-A aarch64 cores
+ that offer hardware virtualization support, and that are released starting 
+with 2015.
+
+Firecracker may work on other x86 and Arm 64-bit CPUs with support for hardware 
+virtualization, but any such platform is currently not supported and not fit 
+for production. If you want to run Firecracker on such platforms, please 
+[open a feature request](https://github.com/firecracker-microvm/firecracker/issues/new?assignees=&labels=&template=feature_request.md&title=%5BFeature+Request%5D+Title).
+
 ## Performance
 
 Firecracker's performance characteristics are listed as part of the

diff --git a/SPECIFICATION.md b/SPECIFICATION.md
@@ -5,7 +5,7 @@ minimal-overhead execution of container and serverless workloads. These
 specifications are enforced by integration tests (that run for each PR and
 master branch merge).
 
-On an I3.metal instance¹, with hyperthreading disabled and given host system
+On an M5D.metal instance¹, with hyperthreading disabled and given host system
 resources are available (e.g., there are enough free CPU cycles, there is
 enough RAM, etc.), customers can rely on the following:
 
@@ -56,8 +56,8 @@ enough RAM, etc.), customers can rely on the following:
    pipes are full will be lost. Any such events will be signaled through the
    `lost-logs` and `lost-metrics` counters.
 
-¹ I3.metal instances:
-[https://aws.amazon.com/ec2/instance-types/i3/](https://aws.amazon.com/ec2/instance-types/i3/)
+¹ M5D.metal instances:
+[https://aws.amazon.com/ec2/instance-types/m5/](https://aws.amazon.com/ec2/instance-types/m5/)
 
 ² CPU ms are actual ms of a user space thread's on-CPU runtime; useful for
   getting consistent measurements for some performance metrics.

diff --git a/docs/images/version_graph.png b/docs/images/version_graph.png
diff --git a/docs/snapshotting/snapshot-support.md b/docs/snapshotting/snapshot-support.md
@@ -9,6 +9,13 @@ guest workload at that particular point in time.
 
 ## Snapshotting in Firecracker
 
+### Supported platforms
+
+The Firecracker snapshot feature is in [developer preview](docs/RELEASE_POLICY.md) 
+on all CPU micro-architectures listed in [README](../README.md#supported-platforms) 
+except ARM which is not supported.
+
+### Overview
 A Firecracker microVM snapshot can be used for loading it later in a different
 Firecracker process, and the original guest workload is being simply resumed.
 
@@ -31,9 +38,8 @@ flexibility to our snapshotting support. This means that taking a snapshot resul
 in multiple files that are composing the full microVM snapshot:
 - the guest memory file,
 - the microVM state file,
-- zero or more disk files (depending on how many the guest had; these are
-  **managed by the users**, which means they need to externally back up their
-  block devices backing files).
+- zero or more disk files (depending on how many the guest had; these are  
+**managed by the users**).
 
 The design allows sharing of memory pages and read only disks between multiple
 microVMs. When loading a snapshot, instead of loading at resume time the full
@@ -45,7 +51,22 @@ This has the advantage of very fast snapshot loading times, but comes with the c
 of having to keep the guest memory file around for the entire lifetime of the
 resumed microVM.
 
-*Note*: Snapshotting is currently supported only on `x86_64` machines.
+## Performance
+
+The Firecracker snapshot create/resume performance depends on the memory size,
+vCPU count and emulated devices count. The Firecracker CI runs snapshots tests 
+on AWS **m5d.metal** instances and the baseline for snapshot resume latency 
+target is under **8ms** with 5ms p90 for a microvm with this specs: 
+2vCPU/512MB/1 block/1 net device.
+
+## Known issues and limitations
+
+- High snapshot latency on 5.4+ host kernels - 
+[#2129](https://github.com/firecracker-microvm/firecracker/issues/2129)
+- Guest network connectivity is not guaranteed to be preserved after resume
+- Restoring microVMs with vsock devices doesn't work.
+- Poor entropy and replayable randomness when resuming multiple microvms which 
+deal with cryptographic secrets. Please see [Snapshot security and uniqueness](#snapshot-security-and-uniqueness)
 
 ## Firecracker Snapshotting characteristics