stream: socket.setEncoding(null) to receive binary Buffers rather than strings has no effect

[inversion]

• Version: v5.8.0
• Platform: Darwin 15.3.0 Darwin Kernel Version 15.3.0: Thu Dec 10 18:40:58 PST 2015; root:xnu-3248.30.4~1/RELEASE_X86_64 x86_64
• Subsystem: stream

I've tried node 5.8.0, 5.10.0 and 4.4.2. I always receive string objects in my data handler unless I manually remove the decoder from the socket readableState:

const server = net.createServer( ( c ) => {

    // See https://nodejs.org/api/stream.html#stream_readable_setencoding_encoding
    c.setEncoding( null );

    // Hack that must be added to make this work as expected
    delete c._readableState.decoder;

    c.on( 'data', ( d ) => {

        // Expect 'object' but get 'string' without deleting the decoder manually
        console.log( typeof d );
    } );

} );

[inversion]

Perhaps this might be a bug in ReadableStream based on a cursory examination of https://github.com/nodejs/node/blob/master/lib/_stream_readable.js#L193

I haven't contributed to Node before but if it seems like a correct approach to null out the encoder if enc === null in that function then I would be more than happy to submit a PR/tests.

[mscdex]

Doc change happened in #5155.

@TriAnMan @silverwind @jasnell From what I can tell passing null has never disabled an existing StringDecoder instance?

[silverwind]

Hmm, that doc change looks wrong, actually. string_decoder does fallback to 'utf8' when encoding is falsy.

[TriAnMan]

I've using this workaround for v4.3.1 and for v0.10.36
Here is a code snippet:

var processServerResponse = function (data) {
    data.setEncoding(null); // Fix the bug with braked utf8 symbols, converted into two unicode REPLACEMENT CHARACTER

    var str = '';
    data.on('data', function (chunk) {
        str += chunk;
    });
    data.on('end', function () {
        // do something with str
    });
};

http.request({/* some valid http options */}, processServerResponse).end();

I don't know what is a typeof chunk but can test.

[TriAnMan]

If I use data.setEncoding(null) then typeof chunk is a 'string'
If I comment out setEncoding then typeof chunk becomes an 'object'

This is for v4.3.1, can't test v0.10.36 right now.

[jasnell]

@mscdex ... as far as I know, utf8 is the default when setEncoding is null. If calling data.setEncoding('buffer') does not make it return a buffer, then that's a change we should make. We use encoding = 'buffer' in other places to explicitly ask for returning buffers (e.g. in 'fs')

[mscdex]

@jasnell IIRC it does not return buffers using 'buffer' since the value is passed to StringDecoder() which would treat it as a single-byte/binary string encoding.

[timelesshaze]

It doesn't work with 'buffer' because it's not a valid encoding.

Buffer.isEncoding('buffer')
> false

The problem is caused by the falsy check in string_decoder.js as @silverwind already said.
Another quick fix is to use non-flowing mode.

'use strict';
const cp = require('child_process');
const child = cp.spawn('ls', { stdio:['ignore','pipe','ignore'] });

// This works
child.stdout.on('readable', () => {
  let chunk;
  while (null !== (chunk = child.stdout.read())) {
    console.log('read %s', typeof chunk);
  }
});

// Type will always be a string
child.stdout.setEncoding(null);
child.stdout.on('data', (chunk) => {
  console.log('onData %s', typeof chunk);
});

[mhart]

This workaround also works – pretty hacky but does the trick until this is fixed (and backported?):

// Simulate socket.setEncoding(null):
if (socket._readableState) {
  delete socket._readableState.decoder
  delete socket._readableState.encoding
}

EDIT: Sorry, just noticed this is basically in the issue description – my bad for not reading fully!