Allow customers to passthrough azcopy flags in az storage copy command

[amishra-dev]

Is your feature request related to a problem? Please describe.
azcopy has a rich set of options that lets the customer configure many facets of uploads like block size et al. Can we create an option like --passthrough ""
anything within the quote above gets patched through to azcopy as is

Describe the solution you'd like
suggested above

Describe alternatives you've considered
LEverage azcopy environment variable where we can, but not everything is in environment variables

Additional context

[evelyn-ys]

Has planned in this sprint

[evelyn-ys]

Hi @amishra-dev, I have designed --extra-options parameter to take all the pass-through arguments like:

az storage copy -s source_file -d container_url --extra-options "--block-blob-size 0.25 --check-length"

But all the extra_options should be quoted with "" or ''. And this will cause complication if these extra options contain "" or '' themselves. See #20702 (comment)

Do you think azcopy options need "" or '' often? I'd like to know your thoughts on @jiasli's comments.

[evelyn-ys]

Do you mind if I mark the new parameter --extra-options as experimental or preview?

[amishra-dev]

--extra-options as experimental sounds good to me. What do you think @zezha-msft?

[amishra-dev]

Hi @amishra-dev Ashish Mishra (AZURE) FTE, I have designed --extra-options parameter to take all the pass-through arguments like:

az storage copy -s source_file -d container_url --extra-options "--block-blob-size 0.25 --check-length"

But all the extra_options should be quoted with "" or ''. And this will cause complication if these extra options contain "" or '' themselves. See #20702 (comment)

Do you think azcopy options need "" or '' often? I'd like to know your thoughts on @jiasli Jiashuo Li FTE's comments.

@zezha-msft would be the best person to answer that.

[zezha-msft]

Quotes might be needed for some flags where the user is free to put any content, some examples:

--metadata string Upload to Azure Storage with these key-value pairs as metadata.
--include-pattern string Include only these files when copying. This option supports wildcard characters (). Separate files by using a ;.
--include-regex string Include only the relative path of the files that align with regular expressions. Separate regular expressions with ';'.
--include-path string Include only these paths when copying. This option does not support wildcard characters (). Checks relative path prefix (For example: myFolder;myFolder/subDirName/file.pdf).
--include-attributes string (Windows only) Includes files whose attributes match the attribute list. For example: A;S;R

@evelyn-ys is it possible to instead treat any unspecified flag as an extra flag for AzCopy? For example, the user could type:

az storage copy -s source_file -d container_url --bla="bla" --foo="foo" --bar="bar"

In this case, the 3 flags bla/foo/bar are all undefined by the cli, so they are passed directly to AzCopy. If they are bad flags and don't work, AzCopy will throw the right error.

[evelyn-ys]

@evelyn-ys is it possible to instead treat any unspecified flag as an extra flag for AzCopy?

I'm afraid not. CLI core inherits the parser from argparse and it will throw exceptions for any unspecified flag:

unrecognized arguments: --bla=bla --foo=foo --bar=bar

But good news is we already have --include-pattern and --include-path supported. Will there be more and more flags need quotes? Do you think it's workable for AzCLI to add only quotes-needed flags and use --extra-options to support other new flags? @zezha-msft

[jiasli]

As I provided in #20702, including quotes in --extra-options's value can become very complex in different shells.

This topic was intensively discussed before in #18869.

Then I would propose the -- solution:

az storage copy -s source_file -d container_url -- --bla="bla" --foo="foo" --bar="bar"

This is a common convention in Linux world:

• https://unix.stackexchange.com/questions/11376/what-does-double-dash-mean
• https://stackoverflow.com/questions/13321458/meaning-of-git-checkout-double-dashes

[zezha-msft]

I like @jiasli's proposal, that does look like an elegant solution.