Running the tests locally

[conorsch]

Perhaps I'm missing something obvious, but I'm unable to run the inspec test suites locally with virtualbox hosts. Creating and converging is fine, but the inspec tests are skipped altogether:

undle exec kitchen verify debian-8
-----> Starting Kitchen (v1.5.0)
-----> Verifying <ansible-19-debian-8>...


Finished in 0.00025 seconds (files took 0.95551 seconds to load)
0 examples, 0 failures

       Finished verifying <ansible-19-debian-8> (0m0.80s).
-----> Verifying <ansible-latest-debian-8>...


Finished in 0.00019 seconds (files took 1.51 seconds to load)
0 examples, 0 failures

       Finished verifying <ansible-latest-debian-8> (0m0.55s).
-----> Kitchen is finished. (0m1.75s)
zlib(finalizer): the stream was freed prematurely.
zlib(finalizer): the stream was freed prematurely.

This is likely a regression introduced by #56, since the integration suites are no longer called "default". Changing the test dir seems to fix it:

bundle exec kitchen verify ansible-latest-debian-8         
-----> Starting Kitchen (v1.5.0)
-----> Verifying <ansible-latest-debian-8>...
.................FFF..........FFF..............................................................

Possible solutions are renaming the inspec test directory, which I don't like, or adding a verifier parameter to the suite blocks to point to the appropriate tests (since the tests are shared across all versions of ansible).

[rndmh3ro]

Thanks for the bug report!
I don't like renaming the inspec-test directory either.
Do you know of any way to add the path as a verifier-parameter?
Otherwise I'll create a feature request on the kitchen-inspec verifier.

[conorsch]

Do you know of any way to add the path as a verifier-parameter?

Drat, I was hoping you did! After two brief forays into the kitchen-inspec source, I'm confident it's not supported, and I don't really think it's worthwhile to submit a change to core if we can work around it.

Therefore we're left with the rather kludgy solution of committing symlinks to version control: dev-sec/ssh-baseline#55

Looks like I was a bit hasty in suggesting the dual-suites approach for supporting multiple Ansible versions, but at present I still think it's worth maintaining 1.9.x and 2.x suites since many folks are dragging their feet on migrating (not that I blame them). Hopefully in a few months it'll be realistic to require v2 for the role, and the 1.9 suite can be removed entirely.

[rndmh3ro]

I don't really like the symlink workaround either so I asked to let the user change the search-path.

[chris-rock]

@conorsch and @rndmh3ro In test-kitchen, the path is always determined by the suite name e.g in your case, the suite name is ansible_1.9 and ansible_latest. Therefore test-kitchen (and this is not restricted to kitchen-inspec) automatically determines the directory with the following pattern: test/integration/%suitename%.

The symlink solution is not the right approach going forward, instead I started to transform test-os-hardening and test-ssh-hardening into proper InSpec profiles. This makes the tests completely independent from the provisioner. At this point of time, its available in the 2.0 branch https://github.com/hardening-io/tests-ssh-hardening/tree/2.0.

I started the work in chef-os-hardening and used kitchen-dokken https://github.com/hardening-io/chef-os-hardening/blob/chris-rock/integration/.kitchen.yml. Next step is to use the new remote profile feature in kitchen-inspec:

suites:
  - name: contains_inspec
    run_list:
      - recipe[os-hardening::default]
    verifier:
      inspec_tests:
        - https://github.com/hardening-io/tests-os-hardening/tree/2.0

In addition I already fixed test-kitchen test-kitchen/test-kitchen#944 to make kitchen-docker work with kitchen-inspec, which is required to use docker, ansible and travis at the same time.

Solution going forward:

• use kitchen-docker (allows us to continuously test all operating systems in travis)
• use ansible provisioner
• use remote SSH InSpec profile

[rndmh3ro]

With merging of #71 locally testing is now fully supported with Ansible 1.9 and Ansible 2 including inspec tests!
If you want to test this @conorsch, you can do so now.

[conorsch]

@rndmh3ro Tested with the VirtualBox backend—works great!