Migrate Unit Tests to use DataRow

[josesimoes]

Description

• Migrate UnitTestFormat.
• Update test framework sub-module.

Motivation and Context

• Addressing Migrate Unit Tests to DataRow Home#1158.

How Has This Been Tested?

Screenshots

Types of changes

• Improvement (non-breaking change that improves a feature, code or algorithm)
• Bug fix (non-breaking change which fixes an issue with code or algorithm)
• New feature (non-breaking change which adds functionality to code)
• Breaking change (fix or feature that would cause existing functionality to change)
• Config and build (change in the configuration and build system, has no impact on code or features)
• Dependencies (update dependencies and changes associated, has no impact on code or features)
• Unit Tests (add new Unit Test(s) or improved existing one(s), has no impact on code or features)
• Documentation (changes or updates in the documentation, has no impact on code or features)

Checklist:

• My code follows the code style of this project (only if there are changes in source code).
• My changes require an update to the documentation (there are changes that require the docs website to be updated).
• I have updated the documentation accordingly (the changes require an update on the docs in this repo).
• I have read the CONTRIBUTING document.
• I have tested everything locally and all new and existing tests passed (only if there are changes in source code).
• I have added new tests to cover my changes.

[josesimoes]

@torbacz I couldn't resist and I've started the migration of the Unit Tests to use the shiny new DataRow 😉

The thing is that it's failing...

Care to take a look please?

[torbacz]

@josesimoes
Will do, probably on Monday.

[torbacz]

@josesimoes
I think I found the problem. Test runner is returning data in "StringFormat_02 - params[...], 123" format and there is a comma in that format. Also in your parameters, there is a comma in one of parameter. I've run into similar issue in one of my own nano project.
So we need to new way to identify DataRow individual test cases (currently done by passed parameters converted to string). I was thinking about doing it by order, by then we will lose easy way to identify which test is failing (new name would be ex. StringFormat_02_1). I need to find a new way to implement that.

Also I found out 2 more issues:

1. When using long string data - it's failing
2. When we add reference into "NFApp" project (the one containing static main method) the test runner is invoking main program instead of tests.

On Monday I'm going to create all 3 issues and try to resolve the first one (which is most important for now)

[josesimoes]

OK... I'm not looking at the code right now so I have little advice to offer at this time. Maybe we can switch the separator from a comma to another char less likely to be found im those contexts like a pipe | or an asterisk * or a #.

[torbacz]

@josesimoes
Is there a bug in string format? I've copied test from your code and it's failing on assertion.

[DataRow("Right align in 10 chars: {0,10:N2}: and then more", 1234.5641, "Right align in 10 chars: 1,234.56 and then more")]
public void StringFormat_02(string formatString, double value, string outcomeMessage)
{
    Console.WriteLine(formatString);
    Console.WriteLine(outcomeMessage);

    var stringFormatValue = string.Format(formatString, value);
    Console.WriteLine(stringFormatValue);

    // Test alignment operator which is the "," and a number. Negative is right aligned, positive left aligned
    Assert.Equal(stringFormatValue, outcomeMessage);
}

Output from console:

Message: 
Right align in 10 chars:   1,234.56: and then more is not equal to Right align in 10 chars: 1

  Standard Error: 
Right align in 10 chars: {0,10:N2}: and then more
Right align in 10 chars: 1,234.56 and then more
Right align in 10 chars:   1,234.56: and then more
    ++++ Exception System.Exception - 0x00000000 (1) ++++
    ++++ Message: Right align in 10 chars:   1,234.56: and then more is not equal to Right align in 10 chars: 1,234.56 and then
 more. 
    ++++ nanoFramework.TestFramework.Assert::Equal [IP: 002f] ++++
    ++++ nanoFramework.TestFramework.Test.TestOfDataRow::StringFormat_02 [IP: 0028] ++++
    ++++ System.Reflection.MethodBase::Invoke [IP: 0000] ++++
    ++++ nanoFramework.TestFramework.UnitTestLauncher::RunTest [IP: 0070] ++++
    ++++ nanoFramework.TestFramework.UnitTestLauncher::Main [IP: 0093] ++++

Note: Additional spaces and :

[josesimoes]

@josesimoes Is there a bug in string format? I've copied test from your code and it's failing on assertion.

Not that I'm aware of... 😯 unit test have been passing in mscorlib.
You can check that easily if you temporarily revert these change on StringFormat and DataRow and run the previous version.

[torbacz]

Found it, when coping from github it removed 2 spaces from parameters....

PR for new version of test framework on the way - resolving ',' in parameters, long strings in parameters and skipping "TestMethod" if "DataRow" is used.

[torbacz]

@josesimoes

Working now :)

[josesimoes]

@torbacz brilliant! Thank you very much for the quick turn around.
Care to commit your changes to the [update-test-framework] branch? seems a waste not doing it... 😉

[torbacz]

@josesimoes
Can't, got 403 error although I've joined nano organization

Remote: Permission to nanoframework/CoreLibrary.git denied to torbacz.
Error encountered while pushing to the remote repository: Git failed with a fatal error.
Git failed with a fatal error.
unable to access 'https://github.com/nanoframework/CoreLibrary/': The requested URL returned error: 403

[torbacz]

I've change only 2 things in your code:

• Submodule updated to latest master (why it's event there? Wouldn't It be easier just to use package?)
• Update test nuget package via VS

[josesimoes]

@torbacz I had to wait for you to join the organization before assigning the permissions. You can now push that branch.

[torbacz]

@josesimoes
Still rejected, this time

remote: error: GH006: Protected branch update failed for refs/heads/update-test-framework.        
remote: error: At least 1 approving review is required by reviewers with write access.        
error: failed to push some refs to 'https://github.com/nanoframework/CoreLibrary'
To https://github.com/nanoframework/CoreLibrary
!	refs/heads/update-test-framework:refs/heads/update-test-framework	[remote rejected] (protected branch hook declined)
Done

[josesimoes]

• Submodule updated to latest master (why it's event there? Wouldn't It be easier just to use package?)

Please DO NOT change that otherwise it wont' work! 😱
This is because this is mscorlib, of which the test framework depends on. It's an "inception" thing.

[josesimoes]

@torbacz it should allow you to push now.

[torbacz]

@josesimoes
Same error.

Btw "Do not change that" - do not change to package or do not update submodule to latest?

[josesimoes]

@josesimoes Same error.

Odd... Please go for a PR against that branch instead.

[josesimoes]

Btw "Do not change that" - do not change to package or do not update submodule to latest?

OK to update sub-module.
NOT OK to switch to package instead of direct reference.

[josesimoes]

@torbacz I'm thinking that it will take a while to complete the migration to DataRow (considering the lack of participation from the community). Therefore I suggest that this is merged as it is and the work can be resumed in future iterations.

[torbacz]

Ok! Maybe create an issue for it?

[josesimoes]

Maybe create an issue for it?

Good idea! It will add visibility to the task. Care to do add it and link it back to this PR?