[WIP] Structuring integration test planning

[tro3]

Taking a shot at #333.

This is a stab at a structure for planning / documenting the integration (e2e) tests. Just started for init.
Start by organizing test_harness directories by command options, in this case:

• init/basic
• init/skip-tools
• init/gopath
• init/special-cases

Under each of those, maintain a README.md file with a table describing the cases - setup and expectations. The tables serve as planning and description of cases for end users.

This is just a beginning of the implementation to show the pattern as well as dive a little deeper on init/basic, but until the new specs finalize, we really can't implement too much. Ideally, the test cases will fall out of the new specs.

Comments welcome (and encouraged).

[tro3]

Note that the "Passing" column requires some explanation. Travis wil pass the tests now, as the cases have been -updated with the results, but those results may not match what is expected in the table. A test is considered to truly pass when the table expected results, the final files, and Travis all line up with a Pass.

[tro3]

@sdboyer any comment here? If it is too early for this stuff, happy to pull it out.

[sdboyer]

First, in general yes, I think this would need to wait until after the re-spec of the commands.

tbh, I'm unsure about this. The categories of basic, skip-tools, gopath, and special-cases don't seem mutually exclusive to me, and I wonder how possible it'll be to come up with categories that really are. And clear, mutually exclusive categories matter here, because they're structuring a filesystem - we can't put a test case in more than one category.

They're also not necessarily obvious; does basic mean "without flags," as opposed to the following two? What about when both skip-tools and gopath are set - that's the mutual exclusivity problem again. I'm also not sure what qualifies as a special case.

Overall, I think this may impose more structure than we can meaningfully use, at least for now. If we actually get to the point of exhaustively testing combinations of possible inputs, then I could see adding more structure. However, when we get to that point, I'd probably prefer to see that expressed somewhere more compactly - e.g., multiple cases within a testcase.json that share an input and (possibly) output - testing combinations of states grows fast, and needing whole directory structures for each one would be onerous.

[sdboyer]

nit: standard practice (as defined by git itself) is to abbreviate hex-encoded SHA1 to first 7 characters. Best to follow it; my mind doesn't immediately parse this as a commit hash because it's 5 instead of 7, and I have to mentally recheck it.

[tro3]

Okay, shutting down for now