Feature Enhancement : Add prefix or suffix to file name while the file is getting streamed via SFTP

[sahilmalhotra92]

Enhancement

I wanted to give a temporary filename to the file placed in remote server while it was getting streamed. I did not have any other option except renaming the file before and after reading the file. It will be very helpful to have a built-in feature to temporary rename file while streaming it.

Gary Russell suggestion to raise issue
https://stackoverflow.com/questions/56470946/add-prefix-or-suffix-to-filename-while-streaming-the-file-using-get-gateway/56478407?noredirect=1#comment99561801_56478407

Code which I tried

     <dependency>
            <groupId>org.springframework.integration</groupId>
            <artifactId>spring-integration-sftp</artifactId>
            <version>5.1.0.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.integration</groupId>
            <artifactId>spring-integration-core</artifactId>
            <version>5.0.6.RELEASE</version>
        </dependency>

Sftp.outboundGateway(sftpSessionFactory(), GET, "payload.remoteDirectory + payload.filename").options(STREAM)

[artembilan]

I wonder if this is really going to be a robust solution since we just might not have MV permission on that SFTP side, only reading one.

I would suggest to consider an Idempotent Receiver solution for such a requirement: https://docs.spring.io/spring-integration/docs/current/reference/html/#idempotent-receiver

This way you won't process the same file again and will have some external tool like Redis to check that file has been processed.

If you need to release a file in the end after streaming, you are good to remove an entry from the ConcurrentMetadataStore for that processed file.

Although I might not mind against MV solution if that is pretty simple and really brings enough value over existing approaches.

Does it make sense?

[garyrussell]

I think the issue is if there are multiple instances. The rename would prevent a second instance from finding a file that is already being fetched by another instance.

[artembilan]

Right. Isn't that what we do with distributed MetadataStore in source filters and in the idempotent receiver as well?
What is the point to fetch the same file in another instance if it has been processed already?

Also it looks like with the -stream option we don't have a way to to rename file back automatically, unless we are going to abuse .setHeader(IntegrationMessageHeaderAccessor.CLOSEABLE_RESOURCE, session)...

I'm not against the feature, I just want to see if we indeed need it so strongly over and existing functionality, although I agree that idempotent receiver doesn't sound so obvious and simple here...

[garyrussell]

@sahilmalhotra92 Can you provide the exact use case for why you need this?

Also it looks like with the -stream option we don't have a way to to rename file back automatically, unless we are going to abuse .setHeader(IntegrationMessageHeaderAccessor.CLOSEABLE_RESOURCE, session)...

I don't think it's abuse, that header is defined as a Closeable; there's no reason we can't wrap the session in another Closable.

[garyrussell]

@sahilmalhotra92 Actually, for "advanced" cases like this, perhaps it would be better to just use the SftpRemoteFileTemplate directly in your code instead of the gateway...

template.rename(...);
template.get(pathToFile, inputStream -> ...);
template.rename(...); // or template.remove(...);

WDYT?

[sahilmalhotra92]

@garyrussell Currently I am renaming the file and renaming it once read, it is solving my purpose even. I was thinking it from nice to have feature if it doesn`t increase the overhead. I will surely replace the logic from fetching the file using SftpRemoteFileTemplate instead of the gateway.

[sahilmalhotra92]

Not sure if I need to close it or leave it like this.

[garyrussell]

We can leave it open; I will put it on the backlog and we can revisit later if there is demand for it.