Huge memory usage when importing large quantities of products

[piotrekkaminski]

imported from: magento/magento2#10914

When importing huge amounts of products Magento 2 is loading contents of importexport_importdata to memory and then iterate it in memory which results in huge memory usage

Preconditions

1. Magento 2.1.7
2. CSV with 1 500 000 products

Steps to reproduce

1. Validate 1 500 000 products
2. Run import

Expected result

1. Memory usage for iterating trough import data should be constant

Actual result

1. Whole table contents is loaded to memory

[piotrekkaminski]

Pull request: magento/magento2#10918

[piotrekkaminski]

_From @magento-engcom-team: @diwipl, thank you for your report.
We've created internal ticket(s) MAGETWO-84592 to track progress on the issue. _