[compliance_checker] add wegihts_initialiazation rule

[mwawrzos]

This is an update to the compliance checker that introduces tests against the rules described in #80.

Update for the following models is ready:

• recommendation
• object_detection
• image_segmentation
• rnn_speech_recognition (it was ready before)
• language_model
• single_stage_detector
• image_classification
• reinforcement

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[johntran-nv]

@bitfort and @petermattson , what do you think is an appropriate approval list for this one? I believe most folks are already aware of this requirement as we talked about it in the SWG, and folks even brought it up in other topics, so I'm inclined to just accept, but I think it would be good to get some other eyes on it.

[petermattson]

We should probably send a "summary of logging changes, final warning for comment" to the Training WG mailing, and summarize this, RCP checking etc. Key thing here IMO is not to surprise people, more than we expect folks to have major comments. After a decent period, we do a normal review for finer points. WDYT?

…

On Wed, Apr 21, 2021 at 11:20 AM johntran-nv ***@***.***> wrote: @bitfort <https://github.com/bitfort> and @petermattson <https://github.com/petermattson> , what do you think is an appropriate approval list for this one? I believe most folks are already aware of this requirement as we talked about it in the SWG, and folks even brought it up in other topics, so I'm inclined to just accept, but I think it would be good to get some other eyes on it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#92 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIIVUHO467BC7LNEMLG3PULTJ4JOJANCNFSM4ZVU3ZLQ> .

[johntran-nv]

Good idea, Peter. I sent out an email to the training alias, and said that I'd come back and merge on Friday, if no one objects.

[emizan76]

A couple of issues / questions just came up:

1. Has weight initialization logging been added to the references? I see only RNNT and Unet3D supporting that. Or the reference implementations do not have to follow this rule? I see a log and compliance directory in the training repo but I do not know how actively they are maintained and which submission rules are followed by the references.
2. Is there an assumption that tensor names should be the same as the reference, or the ones listed in this PR? If submissions use different tensor names then it is tedious work to make this logging.

[mwawrzos]

Answering the questions:

1. I was trying to do this way, so references are updated first, and the compliance checker is updated accordingly, but the submitters WG committee was busy with other topics, and there was no time to bring references update topic. @TheKanter said on one of the benchmark infra WG meetings, that contractors will be hired, and they can update references. Apart from that, most of the references are outdated now, some of them don't even support MLPerf logging. That is why I decided to propose changes this way.
2. Tensor names were supposed to be straightforward. I'm open to change them. If some benchmark is described in too much detail, or too little detail, it is also subject to change. The intention is to make submission review easier, so finding sources initializing weights is simpler. Does tensor name change to reference-framework-like make it more acceptable?

[petermattson]

Folks, should we maybe consider pushing this out a version until we can at least implement and test in the references? Aligning tensor names is a *lot* of work, and tensors for some of the benchmarks may be *very* big. Not sure we want to bloat logging files that much.

…

On Thu, Apr 22, 2021 at 2:13 AM Marek Wawrzos ***@***.***> wrote: Answering the questions: 1. I was trying to do this way, so references are updated first, and the compliance checker is updated accordingly, but the submitters WG committee was busy with other topics, and there was no time to bring references update topic. @TheKanter <https://github.com/TheKanter> said on one of the benchmark infra WG meetings, that contractors will be hired, and they can update references. Apart from that, most of the references are outdated now, some of them don't even support MLPerf logging. That is why I decided to propose changes this way. 2. Tensor names were supposed to be straightforward. I'm open to change them. If some benchmark is described in too much detail, or too little detail, it is also subject to change. The intention is to make submission review easier, so finding sources initializing weights is simpler. Does tensor name change to reference-framework-like make it more acceptable? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#92 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIIVUHLVHBVZIJKESIAFE73TJ7SETANCNFSM4ZVU3ZLQ> .

[johntran-nv]

That's a good suggestion, Peter. Let's defer this to v1.1, then. I'll leave it open.

[petermattson]

Thanks Jon!

…

On Fri, Apr 23, 2021 at 10:57 AM johntran-nv ***@***.***> wrote: That's a good suggestion, Peter. Let's defer this to v1.1, then. I'll leave it open. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#92 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIIVUHNZCJ4HYCMGMWN2HSLTKGYI7ANCNFSM4ZVU3ZLQ> .

[xyhuang]

Fixed in #153