aarch64: enable coverage integration test #2090

dianpopa · 2020-08-20T11:03:21Z

Reason for This PR

Fixes #1579.

Description of Changes

Read commits one by one.

This functionality can be added in rust-vmm.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license.

PR Checklist

[Author TODO: Meet these criteria.]
[Reviewer TODO: Verify that these criteria are met. Request changes if not]

All commits in this PR are signed (git commit -s).
The reason for this PR is clearly provided (issue no. or explanation).
The description of changes is clear and encompassing.
Any required documentation changes (code and docs) are included in this PR.
Any newly added unsafe code is properly documented.
Any API changes are reflected in firecracker/swagger.yaml.
Any user-facing changes are mentioned in CHANGELOG.md.
All added/changed functionality is tested.

acatangiu

LGTM 👍

dianpopa · 2020-10-07T16:57:41Z

Due to observed variances for coverage on aarch64, we decided not to merge this functionaltiy. Howver, I did some digging and ran the coverage test for aarch64 20 times in a row, and the reported coverage was exactly the same each time. I posted the output in the file attached.

ubuntu@dpopa-arm:/repos/firecracker/tools/devctr$ ../devtool shell -p
[Firecracker devtool] Dropping to a privileged shell prompt ...
[Firecracker devtool] Note: /repos/firecracker is bind-mounted under /firecracker
[Firecracker devtool] You are running as root; any files that get created under /firecracker will be owned by root.
root@fe841bced6db:/firecracker# cd tests/
root@fe841bced6db:/firecracker/tests# for i in {1..20}; do
> pytest integration_tests/build/test_coverage.py::test_coverage                      
> done
================================================================== test session starts ==================================================================
platform linux -- Python 3.6.9, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
cachedir: ../build/pytest_cache
rootdir: /firecracker/tests, configfile: pytest.ini
plugins: timeout-1.4.2
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 1 item                                                                                                                                        

[ unsafe | 1 tests | 1 worker(s) ]
  FAILED    integration_tests/build/test_coverage.py::test_coverage

======================================================================= FAILURES ========================================================================
_____________________________________________________________________ test_coverage _____________________________________________________________________
integration_tests/build/test_coverage.py:152: in test_coverage
    assert coverage - coverage_target_pct <= COVERAGE_MAX_DELTA,\
E   AssertionError: Current code coverage (82.14%) is above the target (81.95%).
E     Please update the value of COVERAGE_DICT.
E   assert (82.13695612263632 - 81.95) <= 0.05
----------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------
Number of executable lines: 10894
Number of covered lines: 8948
Thus, coverage is: 82.14%
================================================================ short test summary info ================================================================
FAILED integration_tests/build/test_coverage.py::test_coverage - AssertionError: Current code coverage (82.14%) is above the target (81.95%).
============================================================= 1 failed in 93.36s (0:01:33) ==============================================================
================================================================== test session starts ==================================================================
platform linux -- Python 3.6.9, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
cachedir: ../build/pytest_cache
rootdir: /firecracker/tests, configfile: pytest.ini
plugins: timeout-1.4.2
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 1 item                                                                                                                                        

[ unsafe | 1 tests | 1 worker(s) ]
  FAILED    integration_tests/build/test_coverage.py::test_coverage

======================================================================= FAILURES ========================================================================
_____________________________________________________________________ test_coverage _____________________________________________________________________
integration_tests/build/test_coverage.py:152: in test_coverage
    assert coverage - coverage_target_pct <= COVERAGE_MAX_DELTA,\
E   AssertionError: Current code coverage (82.14%) is above the target (81.95%).
E     Please update the value of COVERAGE_DICT.
E   assert (82.13695612263632 - 81.95) <= 0.05
----------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------
Number of executable lines: 10894
Number of covered lines: 8948
Thus, coverage is: 82.14%
================================================================ short test summary info ================================================================
FAILED integration_tests/build/test_coverage.py::test_coverage - AssertionError: Current code coverage (82.14%) is above the target (81.95%).
============================================================= 1 failed in 94.55s (0:01:34) ==============================================================
================================================================== test session starts ==================================================================
platform linux -- Python 3.6.9, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
cachedir: ../build/pytest_cache
rootdir: /firecracker/tests, configfile: pytest.ini
plugins: timeout-1.4.2
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 1 item                                                                                                                                        

[ unsafe | 1 tests | 1 worker(s) ]
  FAILED    integration_tests/build/test_coverage.py::test_coverage

======================================================================= FAILURES ========================================================================
_____________________________________________________________________ test_coverage _____________________________________________________________________
integration_tests/build/test_coverage.py:152: in test_coverage
    assert coverage - coverage_target_pct <= COVERAGE_MAX_DELTA,\
E   AssertionError: Current code coverage (82.14%) is above the target (81.95%).
E     Please update the value of COVERAGE_DICT.
E   assert (82.13695612263632 - 81.95) <= 0.05
----------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------
Number of executable lines: 10894
Number of covered lines: 8948
Thus, coverage is: 82.14%
================================================================ short test summary info ================================================================
FAILED integration_tests/build/test_coverage.py::test_coverage - AssertionError: Current code coverage (82.14%) is above the target (81.95%).
============================================================= 1 failed in 94.04s (0:01:34) ==============================================================
================================================================== test session starts ==================================================================
platform linux -- Python 3.6.9, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
cachedir: ../build/pytest_cache
rootdir: /firecracker/tests, configfile: pytest.ini
plugins: timeout-1.4.2
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 1 item                                                                                                                                        

[ unsafe | 1 tests | 1 worker(s) ]
  FAILED    integration_tests/build/test_coverage.py::test_coverage

======================================================================= FAILURES ========================================================================
_____________________________________________________________________ test_coverage _____________________________________________________________________
integration_tests/build/test_coverage.py:152: in test_coverage
    assert coverage - coverage_target_pct <= COVERAGE_MAX_DELTA,\
E   AssertionError: Current code coverage (82.14%) is above the target (81.95%).
E     Please update the value of COVERAGE_DICT.
E   assert (82.13695612263632 - 81.95) <= 0.05
----------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------
Number of executable lines: 10894
Number of covered lines: 8948
Thus, coverage is: 82.14%
================================================================ short test summary info ================================================================
FAILED integration_tests/build/test_coverage.py::test_coverage - AssertionError: Current code coverage (82.14%) is above the target (81.95%).
============================================================= 1 failed in 93.50s (0:01:33) ==============================================================
================================================================== test session starts ==================================================================
platform linux -- Python 3.6.9, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
cachedir: ../build/pytest_cache
rootdir: /firecracker/tests, configfile: pytest.ini
plugins: timeout-1.4.2
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 1 item                                                                                                                                        

[ unsafe | 1 tests | 1 worker(s) ]

  FAILED    integration_tests/build/test_coverage.py::test_coverage

======================================================================= FAILURES ========================================================================
_____________________________________________________________________ test_coverage _____________________________________________________________________
integration_tests/build/test_coverage.py:152: in test_coverage
    assert coverage - coverage_target_pct <= COVERAGE_MAX_DELTA,\
E   AssertionError: Current code coverage (82.14%) is above the target (81.95%).
E     Please update the value of COVERAGE_DICT.
E   assert (82.13695612263632 - 81.95) <= 0.05
----------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------
Number of executable lines: 10894
Number of covered lines: 8948
Thus, coverage is: 82.14%
================================================================ short test summary info ================================================================
FAILED integration_tests/build/test_coverage.py::test_coverage - AssertionError: Current code coverage (82.14%) is above the target (81.95%).
============================================================= 1 failed in 93.47s (0:01:33) ==============================================================
================================================================== test session starts ==================================================================
platform linux -- Python 3.6.9, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
cachedir: ../build/pytest_cache
rootdir: /firecracker/tests, configfile: pytest.ini
plugins: timeout-1.4.2
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 1 item                                                                                                                                        

[ unsafe | 1 tests | 1 worker(s) ]
  FAILED    integration_tests/build/test_coverage.py::test_coverage

======================================================================= FAILURES ========================================================================
_____________________________________________________________________ test_coverage _____________________________________________________________________
integration_tests/build/test_coverage.py:152: in test_coverage
    assert coverage - coverage_target_pct <= COVERAGE_MAX_DELTA,\
E   AssertionError: Current code coverage (82.14%) is above the target (81.95%).
E     Please update the value of COVERAGE_DICT.
E   assert (82.13695612263632 - 81.95) <= 0.05
----------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------
Number of executable lines: 10894
Number of covered lines: 8948
Thus, coverage is: 82.14%
================================================================ short test summary info ================================================================
FAILED integration_tests/build/test_coverage.py::test_coverage - AssertionError: Current code coverage (82.14%) is above the target (81.95%).
============================================================= 1 failed in 93.93s (0:01:33) ==============================================================
================================================================== test session starts ==================================================================
platform linux -- Python 3.6.9, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
cachedir: ../build/pytest_cache
rootdir: /firecracker/tests, configfile: pytest.ini
plugins: timeout-1.4.2
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 1 item                                                                                                                                        

[ unsafe | 1 tests | 1 worker(s) ]



  FAILED    integration_tests/build/test_coverage.py::test_coverage

======================================================================================================= FAILURES ========================================================================================================
_____________________________________________________________________________________________________ test_coverage _____________________________________________________________________________________________________
integration_tests/build/test_coverage.py:152: in test_coverage
    assert coverage - coverage_target_pct <= COVERAGE_MAX_DELTA,\
E   AssertionError: Current code coverage (82.14%) is above the target (81.95%).
E     Please update the value of COVERAGE_DICT.
E   assert (82.13695612263632 - 81.95) <= 0.05
------------------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------------------
Number of executable lines: 10894
Number of covered lines: 8948
Thus, coverage is: 82.14%
================================================================================================ short test summary info ================================================================================================
FAILED integration_tests/build/test_coverage.py::test_coverage - AssertionError: Current code coverage (82.14%) is above the target (81.95%).
============================================================================================= 1 failed in 93.97s (0:01:33) ==============================================================================================
================================================================================================== test session starts ==================================================================================================
platform linux -- Python 3.6.9, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
cachedir: ../build/pytest_cache
rootdir: /firecracker/tests, configfile: pytest.ini
plugins: timeout-1.4.2
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 1 item                                                                                                                                                                                                        

[ unsafe | 1 tests | 1 worker(s) ]
  FAILED    integration_tests/build/test_coverage.py::test_coverage

======================================================================================================= FAILURES ========================================================================================================
_____________________________________________________________________________________________________ test_coverage _____________________________________________________________________________________________________
integration_tests/build/test_coverage.py:152: in test_coverage
    assert coverage - coverage_target_pct <= COVERAGE_MAX_DELTA,\
E   AssertionError: Current code coverage (82.14%) is above the target (81.95%).
E     Please update the value of COVERAGE_DICT.
E   assert (82.13695612263632 - 81.95) <= 0.05
------------------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------------------
Number of executable lines: 10894
Number of covered lines: 8948
Thus, coverage is: 82.14%
================================================================================================ short test summary info ================================================================================================
FAILED integration_tests/build/test_coverage.py::test_coverage - AssertionError: Current code coverage (82.14%) is above the target (81.95%).
============================================================================================= 1 failed in 93.58s (0:01:33) =============================================================================================

@firecracker-microvm/compute-capsule please re-review!

Signed-off-by: Diana Popa <[email protected]>

Fixes firecracker-microvm#1579. Signed-off-by: Diana Popa <[email protected]>

For a full description, checkout Github issue 2216. Signed-off-by: Diana Popa <[email protected]>

AlexandruCihodaru

LGTM. 🚀

dianpopa added Feature: CPU Support: ARM Status: Blocked Indicates that an issue or pull request cannot currently be worked on labels Aug 20, 2020

dianpopa added this to the Feature parity on aarch64 milestone Aug 20, 2020

dianpopa self-assigned this Aug 20, 2020

dianpopa requested a review from a team August 20, 2020 11:03

dianpopa force-pushed the little_fixes_3 branch from 6df1520 to 90ac2ac Compare August 20, 2020 11:49

dianpopa added Status: Awaiting review Indicates that a pull request is ready to be reviewed and removed Status: Blocked Indicates that an issue or pull request cannot currently be worked on labels Aug 20, 2020

dianpopa force-pushed the little_fixes_3 branch from 90ac2ac to 0efb594 Compare August 20, 2020 13:22

acatangiu previously approved these changes Aug 24, 2020

View reviewed changes

dianpopa dismissed acatangiu’s stale review via b3250c3 August 24, 2020 13:39

dianpopa force-pushed the little_fixes_3 branch from 0efb594 to b3250c3 Compare August 24, 2020 13:39

dianpopa force-pushed the little_fixes_3 branch from b3250c3 to 65cb119 Compare October 7, 2020 16:20

dianpopa force-pushed the little_fixes_3 branch 2 times, most recently from dd34ed8 to bfdbde4 Compare October 7, 2020 17:01

dianpopa requested a review from a team October 7, 2020 17:09

dianpopa force-pushed the little_fixes_3 branch from bfdbde4 to 7b405e8 Compare October 23, 2020 14:23

dianpopa added 4 commits October 26, 2020 18:32

devctr: add cargo kcov for aarch64

8ec7bae

Signed-off-by: Diana Popa <[email protected]>

devtool: use kcov enabled image v21

50752d9

Signed-off-by: Diana Popa <[email protected]>

tests: enable coverage on aarch64

3dfb36a

Fixes firecracker-microvm#1579. Signed-off-by: Diana Popa <[email protected]>

temporarily disable assert for aarch64

5799651

For a full description, checkout Github issue 2216. Signed-off-by: Diana Popa <[email protected]>

dianpopa force-pushed the little_fixes_3 branch from 7b405e8 to 5799651 Compare October 26, 2020 16:55

AlexandruCihodaru approved these changes Oct 26, 2020

View reviewed changes

acatangiu approved these changes Oct 27, 2020

View reviewed changes

acatangiu merged commit 323a6c3 into firecracker-microvm:master Oct 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

aarch64: enable coverage integration test #2090

aarch64: enable coverage integration test #2090

Uh oh!

dianpopa commented Aug 20, 2020 •

edited

Loading

Uh oh!

acatangiu left a comment

Uh oh!

dianpopa commented Oct 7, 2020

Uh oh!

AlexandruCihodaru left a comment

Uh oh!

Uh oh!

aarch64: enable coverage integration test #2090

aarch64: enable coverage integration test #2090

Uh oh!

Conversation

dianpopa commented Aug 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reason for This PR

Description of Changes

License Acceptance

PR Checklist

Uh oh!

acatangiu left a comment

Choose a reason for hiding this comment

Uh oh!

dianpopa commented Oct 7, 2020

Uh oh!

AlexandruCihodaru left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dianpopa commented Aug 20, 2020 •

edited

Loading