buffer: add buffer.transcode #9038

jasnell · 2016-10-11T21:16:00Z

Checklist

make -j8 test (UNIX), or vcbuild test nosign (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

buffer

Description of change

Add buffer.transcode(source, from, to) method. Primarily uses ICU to transcode a buffer's content from one of Node.js' supported encodings to another.

Originally part of a proposal to add a new unicode module. Decided to refactor the approach towrds individual PRs without a new module.

Refs: #8075
/cc @trevnorris @addaleax

jasnell · 2016-10-11T21:16:52Z

CI: https://ci.nodejs.org/job/node-test-pull-request/4476/

trevnorris · 2016-10-11T23:35:22Z

lib/internal/buffer.js

what about possibly placing this in lib/internal/buffer-transcode.js and conditionally require()'ing it. purely cosmetic though, to prevent an extra level of indent. or you could just return early. :)

if (!process.binding('config').hasIntl) return;

trevnorris · 2016-10-11T23:36:33Z

lib/internal/buffer.js

We going to have a complaint about not supporting SharedArrayBuffer for this?

Eventually, perhaps. Not too worried about that for now.

jasnell · 2016-10-12T00:03:50Z

Updated

addaleax · 2016-10-11T21:26:51Z

doc/api/buffer.md

Maybe give an example of transcoding is not possible?

addaleax · 2016-10-11T22:54:05Z

src/node_i18n.cc

Hm – this could be a ThrowICUError function, right?

addaleax · 2016-10-11T22:55:53Z

src/node_i18n.cc

This is fine but at some point this might become a member of the MaybeStackBuffer class? I realize that would conflict a bit with the MaybeStackBuffer<UChar> overload, maybe leave a TODO here?

addaleax · 2016-10-12T08:58:22Z

lib/internal/buffer.js

Is that final to residue from editing?

Yeah, slight brain malfunction there I think ;-)

addaleax · 2016-10-12T08:58:44Z

lib/internal/buffer.js

Mhhh this returns a string or a Buffer depending on the target encoding? I don’t think binary-to-text encodings should be allowed here, .toString() is the right method for them.

Yeah, you're right. I'll pull these back out.

jasnell · 2016-10-12T14:35:02Z

@addaleax ... updated! PTAL

addaleax

LGTM, nice!

addaleax · 2016-10-12T21:26:19Z

src/node_i18n.cc

Could this use SwapBytes16?

addaleax · 2016-10-12T21:27:43Z

src/node_i18n.cc

addaleax · 2016-10-12T21:32:49Z

src/node_i18n.cc

The 1024 seem kind of magic here, although I realize that is largely my fault. 😄 (Not sure if there’s anything to do about that)

Should be fixed now!

bnoordhuis · 2016-10-13T05:23:13Z

lib/internal/buffer.js

Style: s/from_enc/fromEncoding/ and s/to_enc/toEncoding/. Ditto for cnv_from and cnv_to.

bnoordhuis · 2016-10-13T05:28:42Z

src/node_i18n.cc

I realize you adapted this code from elsewhere but using snprintf() to format the error message will be much more efficient.

bnoordhuis · 2016-10-13T05:29:47Z

src/node_i18n.cc

Needs a if (e.Empty()) return Local<Value>();.

bnoordhuis · 2016-10-13T05:32:58Z

src/node_i18n.cc

I think this should be if (!ret.Empty()) buf->Release(); - it's leaking memory now when the buffer can't be created.

bnoordhuis · 2016-10-13T05:33:43Z

src/node_i18n.cc

bnoordhuis · 2016-10-13T05:46:49Z

src/node_i18n.cc

Same as above: strict aliasing violation and prone to crashing.

bnoordhuis · 2016-10-13T05:49:22Z

src/node_i18n.cc

It seems like there is ample opportunity to share code between Ucs2FromUtf8 and Utf8FromUcs2, they are 80% identical.

Perhaps. For now I'm more inclined to keep these separate as it makes finding and tweaking bugs a bit easier. I'll take another pass in a separate PR to condense things down.

bnoordhuis · 2016-10-13T05:50:25Z

src/util.h

Shouldn't this also reset length_?

bnoordhuis · 2016-10-13T05:51:54Z

src/util.h

If you move this into a common header, you might want to give it a slightly less generic name; e.g. SPREAD_BUFFER_ARG.

bnoordhuis · 2016-10-13T05:52:37Z

tools/icu/icu-generic.gyp

Just remove the 'defines' block instead of commenting it out.

jasnell · 2016-10-13T15:13:04Z

~~New CI: https://ci.nodejs.org/job/node-test-pull-request/4503/~~
Issues on big endian... trying again: https://ci.nodejs.org/job/node-test-pull-request/4504/

jasnell · 2016-10-13T18:01:47Z

~~Another CI run after cleanups: https://ci.nodejs.org/job/node-test-pull-request/4507/~~ ... that last run was less than successful....
Trying another: https://ci.nodejs.org/job/node-test-pull-request/4508/

jasnell · 2016-10-13T20:40:57Z

CI looks good. @bnoordhuis PTAL... LGTY?

jasnell · 2016-10-17T17:17:12Z

ping @bnoordhuis

trevnorris · 2016-10-17T22:36:07Z

src/node_i18n.cc

if you're returning empty handles as an indicator it's probably be more "V8-ish" have the return signature as a MaybeLocal<Value> instead. been trying to do that in other locations myself.

trevnorris · 2016-10-17T22:39:02Z

src/node_i18n.cc

creating Local<Value>'s but there's no HandleScope. if this callback is expected to always be called within an existing HandleScope (like MakeCallback), mind putting a comment at the top. also like MakeCallback (see src/node.h).

trevnorris · 2016-10-17T22:39:55Z

src/node_i18n.cc

if we know this is a v8::Object then can use e.As<Object>(). that's also more explicit that no extra handle is being created.

trevnorris · 2016-10-17T22:41:13Z

src/node_i18n.cc

The new v8::Maybe<T> API for v8::Object::Set() is annoying and ugly, but if if we're going to use some of the new API might as well use all of it.

trevnorris · 2016-10-17T22:44:43Z

src/node_i18n.cc

if you're manipulating the original memory, why bother take a copy?

trevnorris · 2016-10-17T22:46:50Z

src/node_i18n.cc

if this operation fails, do we want to abort or throw?

trevnorris · 2016-10-17T22:50:08Z

src/node_i18n.cc

for future performance enhancement, detect the alignment of the pointer and perform as many swaps that can be done in a single go.

trevnorris · 2016-10-17T22:53:52Z

lib/internal/buffer.js

i'm a little confused by this whole object, but right here if we're converting ascii to latin1 shouldn't we be passing 'latin1' as the encoding argument to Buffer.from()?

jasnell · 2016-10-18T21:44:18Z

@bnoordhuis ... ok, reworked the implementation with an eye towards simplification and reducing duplication. PTAL
@trevnorris and @addaleax ... if I could trouble each of you to take another look also, I'd appreciate it.

jasnell · 2016-10-21T17:11:03Z

ping @bnoordhuis @trevnorris @addaleax

jasnell · 2016-10-24T17:33:32Z

Thank you for the follow up review @addaleax. Will get this landed tomorrow if there are no further objections.

Add buffer.transcode(source, from, to) method. Primarily uses ICU to transcode a buffer's content from one of Node.js' supported encodings to another. Originally part of a proposal to add a new unicode module. Decided to refactor the approach towrds individual PRs without a new module. Refs: nodejs#8075

jasnell · 2016-10-25T16:12:56Z

New CI run after squashing: https://ci.nodejs.org/job/node-test-pull-request/4665/

jasnell · 2016-10-25T17:10:46Z

green except for unrelated failures. landing

PR updated after view

Add buffer.transcode(source, from, to) method. Primarily uses ICU to transcode a buffer's content from one of Node.js' supported encodings to another. Originally part of a proposal to add a new unicode module. Decided to refactor the approach towrds individual PRs without a new module. Refs: #8075 PR-URL: #9038 Reviewed-By: Anna Henningsen <[email protected]>

jasnell · 2016-10-25T17:13:50Z

Landed in e8eaaa7

Add buffer.transcode(source, from, to) method. Primarily uses ICU to transcode a buffer's content from one of Node.js' supported encodings to another. Originally part of a proposal to add a new unicode module. Decided to refactor the approach towrds individual PRs without a new module. Refs: #8075 PR-URL: #9038 Reviewed-By: Anna Henningsen <[email protected]>

addaleax · 2016-12-21T05:00:51Z

If this is backported to any of the other release lines, it needs to come with #9838

jasnell added the buffer Issues and PRs related to the buffer subsystem. label Oct 11, 2016

nodejs-github-bot added build Issues and PRs related to build files or the CI. c++ Issues and PRs that require attention from people who are familiar with C++. tools Issues and PRs related to the tools directory. labels Oct 11, 2016

jasnell added the semver-minor PRs that contain new features and should be released in the next minor version. label Oct 11, 2016

trevnorris reviewed Oct 11, 2016

View reviewed changes

addaleax reviewed Oct 12, 2016

View reviewed changes

jasnell force-pushed the buffer-transcode branch from 5b65f74 to 62423a0 Compare October 12, 2016 14:18

addaleax approved these changes Oct 12, 2016

View reviewed changes

bnoordhuis requested changes Oct 13, 2016

View reviewed changes

jasnell force-pushed the buffer-transcode branch 3 times, most recently from 9e0464b to 2da087e Compare October 13, 2016 15:11

jasnell force-pushed the buffer-transcode branch from eecad5e to d7340ec Compare October 13, 2016 20:09

trevnorris reviewed Oct 17, 2016

View reviewed changes

src/node_i18n.cc Outdated

Copy link

Contributor

trevnorris Oct 17, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you're manipulating the original memory, why bother take a copy?

trevnorris reviewed Oct 17, 2016

View reviewed changes

src/node_i18n.cc Outdated

Copy link

Contributor

trevnorris Oct 17, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if this operation fails, do we want to abort or throw?

trevnorris reviewed Oct 17, 2016

View reviewed changes

jasnell force-pushed the buffer-transcode branch 3 times, most recently from 6f70820 to 65144bb Compare October 18, 2016 21:43

addaleax approved these changes Oct 21, 2016

View reviewed changes

jasnell force-pushed the buffer-transcode branch from 65144bb to 4d7472b Compare October 25, 2016 16:12

jasnell closed this Oct 25, 2016

srl295 mentioned this pull request Oct 25, 2016

intl: Add more versions from ICU #9266

Closed

4 tasks

evanlucas mentioned this pull request Nov 3, 2016

v7.1.0 proposal - 2016-11-08 #9438

Merged

ChALkeR mentioned this pull request Nov 25, 2016

Feature proposal: buffer.convert(value[, fromEnc], toEnc) #9797

Closed

addaleax mentioned this pull request Dec 21, 2016

buffer: fix transcode for single-byte enc to ucs2 #9838

Closed

3 tasks

MylesBorins added lts-agenda labels Jan 6, 2017

MylesBorins mentioned this pull request Jan 6, 2017

Potential Semver Minor Backports nodejs/Release#177

Closed

9 tasks

MylesBorins added dont-land-on-v4.x and removed lts-watch-v4.x labels Jan 16, 2017

Uh oh!

buffer: add buffer.transcode #9038

buffer: add buffer.transcode #9038

Uh oh!

Conversation

jasnell commented Oct 11, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Affected core subsystem(s)

Description of change

Uh oh!

jasnell commented Oct 11, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jasnell commented Oct 12, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

addaleax Oct 11, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jasnell commented Oct 12, 2016

Uh oh!

addaleax left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jasnell commented Oct 13, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jasnell commented Oct 13, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jasnell commented Oct 13, 2016

Uh oh!

jasnell commented Oct 11, 2016 •

edited

Loading

addaleax Oct 11, 2016 •

edited

Loading

jasnell commented Oct 13, 2016 •

edited

Loading

jasnell commented Oct 13, 2016 •

edited

Loading