Added multiple tokenizers implementing new ITokenizer interface

[ericnewton76]

Issue by Ehryk
Saturday Aug 01, 2015 at 06:05 GMT
Originally opened as gsscoder/commandline#212

---

Persuant to issue #108, I have renamed the default tokenizer to TokenizerGetOpt, and added TokenizerWindows (currently just '/', '//' variant of TokenizerGetOpt), and TokenizerHybrid (allowing both '-','/' and '--','//' together). I also added a ITokenizer interface to set a standard for possibly other tokenizers.

Because static classes cannot implement interfaces, the tokenizers are no longer static and must be instantiated before use (which is now done so using TokenizerGetOpt).

---

Ehryk included the following code: https://github.com/gsscoder/commandline/pull/212/commits

[ericnewton76]

Comment by gsscoder
Saturday Aug 01, 2015 at 06:15 GMT

---

@Ehryk, thanks for PR! I'm not agree to support an hybrid tokenizer but the other parts seem OK.

As you can see I've recently promoted the library from alpha to beta and except fixes I'll delay accepting PR until it stabilizes.

I was about opening an announcement issue on that.

I'll back on the question ASAP.

[ericnewton76]

Comment by gsscoder
Saturday Aug 01, 2015 at 06:20 GMT

---

Please see Issue #213.

[ericnewton76]

Comment by Ehryk
Sunday Aug 02, 2015 at 22:27 GMT

---

Are you comfortable with the general structure (any class that implements the ITokenizer interface can be used to tokenize the command line)?

This way even if you aren't interested in supporting a Hybrid or Windows tokenizer, they could be added and supported separately? There could even be other tokenizers added for variants (of which I am not familiar, but perhaps some other use case would desire).

[ericnewton76]

Comment by gsscoder
Monday Aug 03, 2015 at 05:45 GMT

---

@Ehryk, are you proposing to open this part of the library and let people define their own Tokenizer?

I just want to you don't need an ITokenizer interface since a Func delegate is already its signature:

Func<IEnumerable<string>, IEnumerable<OptionSpecification>, Result<IEnumerable<Token>, Error>>

As used all over the code... :)