Introducing Seed Checker for #85

[shangw-nvidia]

Hi @xyhuang ,

Hope things are going well! Please review this PR for implementing seed checker for #85.

In addition to the rules layout in #85, the following additional requirements are added to root out corner cases:

1. All seeds must be logged through mllog (if choose to log seeds). Any seed logged via any other method will be disgarded.
2. All seeds, if choose to be logged, must be valid integer (convertible via int()).
3. If any run log at lesat one seed, we expect all runs to log at least one seed.
   4. The set of seed(s) that one run logs must be completely different from the set of seed(s) any other run logs.
4. If one run logs one seed on a certain line in a certain source file, no other run can log the same seed on the same line in the same file.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[shangw-nvidia]

Error: committers of Pull Request number 87 have to sign the CLA

I don't know what this means.

[emizan76]

Here, if result files F1 and F2 have seeds S1 and S2, and F1.S1 == F2.S2 this will be marked as a violation even though it is not, right? This case is rare, but do we handle it?

[shangw-nvidia]

By S1 and S2, you meant seeds logged at different lines between different result files?

[emizan76]

Yes, so each file has S1 and S2, and S1 and S2 are the same for different files. Not sure if it is a valid scenario

[shangw-nvidia]

I updated a few new changes to support this.

I think what you pointed out is a good idea in general, however, currently I don't think we have a way to make sure that what being logged matches the source code, thus, seed assertion on a file/lineno granularity might be not rigorous enough. For example, say I launch one run, then I modified something minor in the code (in the sense that I subjectively think a re-run is not really needed) and then launch a second run; even if I use the same seed, the file/lineno is going to be different.

TL;DR: It's probably good for now, but some follow-up work is probably needed to make it more rigorous.