Feature proposal: buffer.convert(value[, fromEnc], toEnc)

[ChALkeR]

While investigating the Buffer usage, I noticed one thing — in many (really, many) cases, Buffers are constructed just for the sake of base64-encoding a string.

base64-encodinng is used in many places, like auth headers, data uris, messages, etc.

With the new API, that would look like Buffer.from(from).toString(encoding).
With the old API, that would look like new Buffer(from).toString(encoding) (note that this lacks checks).

So I propose to add a small utility function, e.g.

Buffer.convert = function(value, targetEncoding, sourceEncoding) {
  if (!Buffer.isEncoding(targetEncoding)) {
    throw new TypeError('"targetEncoding" must be a valid string encoding');
  }
  return Buffer.from(value, sourceEncoding).toString(targetEncoding);
}

Note that the impl does not default to utf-8 and forces an encoding to be specified, this way it would be more clear what the userspace code does.

Sure, that could be done on userside, but this doesn't deserve a separate package, and re-implementing that in all the packages that do conversion also doesn't look very good to me.

Note that if we have had such method earlier, the usage of Buffer without new (and other Buffer-related issues) would have been measurably lower.

If everyone thinks that this is a good idea, I am willing to file a PR for it (impl/docs/tests).

[ChALkeR]

Or perhaps Buffer.convert(value[, sourceEncoding], targetEncoding) would be better — I am not sure of that yet.

[addaleax]

I’d be okay with it, although I would probably prefer it as buffer.convert directly on the buffer module, like we have buffer.transcode (to which convert would be the dual).

And you should probably specify both encodings explicitly. For this conversion to make sense, one of these has to be a binary-to-text encoding like base64, and I don’t think there’s any way of knowing whether it’s supposed to be encoding or decoding?

/cc @nodejs/buffer

[ChALkeR]

@addaleax Ah, I entirely missed that one (from #9038).

Then we could use the same header, like buffer.convert(value, fromEnc, toEnc) and perpahs also just buffer.convert(value, toEnc) (defaulting fromEnc to 'utf-8').

[addaleax]

@ChALkeR I am not sure you can compare the situation wrt default encodings here. For transcode, it makes sense to default to utf-8 for the arguments, because transcode only makes sense to be called for character encodings, not binary-to-text ones.

If you prefer it another way, I guess that’s fine, but I really think explicit encodings would be better here.

[seishun]

Not sure this is a good idea.

• buffer.convert(fromStr, target, source) isn't much simpler than Buffer.from(fromStr, source).toString(target).
• Order of arguments is confusing (I'd expect source encoding to come before target encoding, so it seems different people have different expectations).
• Can be confused with buffer.transcode.
• It might not be clear why it's a part of Buffer API even though both the input and the output are strings.

[ChALkeR]

@seishun Thanks.

• It's a bit shorter, more clear, does the single thing, also it could be optimized later if we decide that it should be.
• I have already changed the order in the title to be buffer.convert(value[, fromEnc], toEnc), I did not update the impl, though. Does that look better?
• We could limit buffer.convert to strings only and remove Buffer support on input. This way, it couldn't be confused with .transcode, which accepts only Buffer objects.

[seishun]

I have already changed the order in the title to be buffer.convert(value[, fromEnc], toEnc), I did not update the impl, though. Does that look better?

I didn't say that the order is better one or the other way. My concern is that people will likely get confused either way.

This way, it couldn't be confused with .transcode, which accepts only Buffer objects.

But they still both live in the buffer module. And it's not obvious from the names which does which.

[Trott]

@ChALkeR Leave this open? (I imagine so, but it's been inactive for long enough that I figure it's worth checking.)

[jasnell]

This is essentially Buffer.transcode()... and can also be accomplished using the Encoding API implementation I've done.

[seishun]

This is essentially Buffer.transcode()

Not quite. Buffer.transcode converts a buffer to a buffer. This converts a string to a string.

[gireeshpunathil]

so the proposal itself has a prototype code, not sure why it got stalled for 18 months. Calling for contributors!