WRKLDS-1191: oc rsync: Add --last flag

[tchap]

The flag can be used to only sync N most recently modified files.

The flag is mutually exclusive to:

• --include
• --exclude
• --delete
• --watch

It is also ignored when the source is a local directory when using tar, because the implementation doesn't allow to select particular files. We would need to exclude all other files, which can explode easily. Simply use a different strategy in that case.

Generally when there is any problem when using --last, the error is ignored and sync happens as if the flag was not specified.

Regarding implementation details, oc rsync performs an extras step when --last is specified, and that is discovering relevant files to select. This is done using manual directory walking when local, for remote the remote executor is used to invoke a shell using find+sort+head.

The resulting filenames are then passed to --files-from for rsync, for tar they are simply passed to the command as arguments.

Tests were added for testing the discovery mechanism, the rest has been tested manually. oc rsync is rather poorly unit-tested in general.

[openshift-ci-robot]

@tchap: This pull request references WRKLDS-1191 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set.

In response to this:

The flag can be used to only sync N most recently modified files.

The flag is mutually exclusive to:

• --include
• --exclude
• --delete
• --watch

It is also ignored when the source is a local directory when using tar, because the implementation doesn't allow to select particular files. We would need to exclude all other files, which can explode easily. Simply use a different strategy in that case.

Generally when there is any problem when using --last, the error is ignored and sync happens as if the flag was not specified.

Regarding implementation details, oc rsync performs an extras step when --last is specified, and that is discovering relevant files to select. This is done using manual directory walking when local, for remote the remote executor is used to invoke a shell using find+sort+head.

The resulting filenames are then passed to --files-from for rsync, for tar they are simply passed to the command as arguments.

Tests were added for testing the discovery mechanism, the rest has been tested manually. oc rsync is rather poorly unit-tested in general.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@tchap: This pull request references WRKLDS-1191 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set.

In response to this:

The flag can be used to only sync N most recently modified files.

The flag is mutually exclusive to:

• --include
• --exclude
• --delete
• --watch

It is also ignored when the source is a local directory when using tar, because the implementation doesn't allow to select particular files. We would need to exclude all other files, which can explode easily. Simply use a different strategy in that case.

Generally when there is any problem when using --last, the error is ignored and sync happens as if the flag was not specified.

Regarding implementation details, oc rsync performs an extras step when --last is specified, and that is discovering relevant files to select. This is done using manual directory walking when local, for remote the remote executor is used to invoke a shell using find+sort+head.

The resulting filenames are then passed to --files-from for rsync, for tar they are simply passed to the command as arguments.

Tests were added for testing the discovery mechanism, the rest has been tested manually. oc rsync is rather poorly unit-tested in general.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tchap
Once this PR has been reviewed and has the lgtm label, please assign ardaguclu for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tchap]

I guess this can also wait for 4.21 as nobody is going to use it in 4.20 anyway...

[tchap]

/test okd-scos-e2e-aws-ovn
/test e2e-metal-ipi-ovn-ipv6
/test e2e-aws-ovn

[tchap]

/retest

[openshift-ci]

@tchap: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.