-
Notifications
You must be signed in to change notification settings - Fork 11
Use temp stream instead of string to buffer content #52
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good to me. i can imagine that this adds some filesystem overhead for medium sized streams where previously things worked in memory and now go to the temporary file, but if people really need to worry on that level they can configure the temp handling, according to php.net
@GrahamCampbell do you agree or are there any doubts?
9b6ff69
to
5b2f677
Compare
Updated by PR to include mechanism to detect available memory and assign maxmemory to the buffer (size of buffer suitable to stay in memory). It would subtract from memory_get_usage() result from PHP I also added a setBufferMaxMemory() public method for overriding this. |
* Update MultipartStreamBuilder::build to buffer built content in a php://temp stream resource instead of (string) $contents. This way, the size of the stream built won't be limited by PHP runtime limit.
* Detect available memory (in bytes). Then config the buffer stream to stay in memory until reaching 1/4 of that. * Added setBufferMaxMemory to manually override that limit.
5b2f677
to
4f5675c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is really cool! i think using max 1/4 of available memory by default is a good idea to not use up all the memory with large streams.
5a8f7e5
to
8ae08b7
Compare
8ae08b7
to
6d0a8d5
Compare
Finally fixed all the style issues :-) |
* Improve documentation to MultipartStreamBuilder::getAvailableMemory
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cheers, looks good to me. i wait until tomorrow so @Nyholm or @GrahamCampbell get a chance to see it before merging.
@yookoala could you please add a changelog note for this improvement?
I've left some comments. There are some issues, some of which are major (but easy to resolve). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you
I agree that we should use a stream, but lets fallback to PHP default behavior.
@Nyholm has suggested some excellent changes. 👍 |
* Remove the memory detection logic from MultipartStreamBuilder::build.
3b0927d
to
f45ef10
Compare
ee50cfc
to
91ec1dc
Compare
* Improve readability of comment. * Use the old __toString approach if the stream is not isReadable().
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good to me, thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool. Im really happy with this now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good work.
in a php://temp stream resource instead of (string) $contents.
This way, the size of the stream built won't be limited by PHP
runtime limit.
What's in this PR?
Rewrite to append stream contents and boundaries into a "php://temp" stream as a buffer. Contents within predefined limit (2MB) will be handled in memory for speed. Otherwise, PHP will create a temp file for storage. This mean the multipart stream size is virtually without size limit (unless if, of course, your harddisk is full).
Why?
The original MultipartStreamBuilder::build would read everything into a string variable
$streams
before making it another stream. If the total size of the content exceed the PHP's memory limit, the process would fail.Checklist