TypeError: value is out of bounds when Insert 1M rows from .sql file

[ty2]

TypeError: value is out of bounds
    at checkInt (buffer.js:825:11)
    at Buffer.writeUInt16LE (buffer.js:883:5)
    at Packet.writeInt16 (/var/www/public/node_modules/mysql2/lib/packets/packet.js:527:15)
    at Packet.writeInt24 (/var/www/public/node_modules/mysql2/lib/packets/packet.js:523:8)
    at Packet.writeHeader (/var/www/public/node_modules/mysql2/lib/packets/packet.js:593:8)
    at Connection.writePacket (/var/www/public/node_modules/mysql2/lib/connection.js:142:10)
    at Query.start (/var/www/public/node_modules/mysql2/lib/commands/query.js:39:14)
    at Query.Command.execute (/var/www/public/node_modules/mysql2/lib/commands/command.js:34:20)
    at Connection.handlePacket (/var/www/public/node_modules/mysql2/lib/connection.js:310:28)
    at Connection.addCommand (/var/www/public/node_modules/mysql2/lib/connection.js:326:10)

any ideas?

[sidorares]

was it accidentally reopened?

[ty2]

i just set the max_allowed_packet = 128M, node-mysql is works fine, but node-mysql2 still shows this error.

here is my code:
https://gist.github.com/terrywong2/56fd19a1ee81d50f77e7

[lillem4n]

Related, when inserting (large) binary blobs I'm getting similar issues:

uncaughtException: value is out of bounds date=Wed Sep 28 2016 10:29:56 GMT+0200 (CEST),
pid=9977,
uid=1005,
gid=1005,
cwd=app,
execPath=/usr/bin/nodejs,
version=v4.4.3,
argv=[
/usr/bin/nodejs,
/usr/lib/node_modules/pm2/lib/ProcessContainerFork.js
],
rss=196702208,
heapTotal=93235296,
heapUsed=89848920,
loadavg=[0.0029296875, 0.0146484375, 0.04541015625],
uptime=13724341,
trace=[
column=11, file=buffer.js, function=checkInt, line=825, method=null, native=false,
column=5, file=buffer.js, function=Buffer.writeUInt16LE, line=883, method=writeUInt16LE, native=false,
column=15, file=app/node_modules/larvitdb/node_modules/mysql2/lib/packets/packet.js, function=Packet.writeInt16, line=707, method=writeInt16, native=false,
column=8, file=app/node_modules/larvitdb/node_modules/mysql2/lib/packets/packet.js, function=Packet.writeInt24, line=703, method=writeInt24, native=false,
column=8, file=app/node_modules/larvitdb/node_modules/mysql2/lib/packets/packet.js, function=Packet.writeHeader, line=791, method=writeHeader, native=false,
column=10, file=app/node_modules/larvitdb/node_modules/mysql2/lib/connection.js, function=PoolConnection.Connection.writePacket, line=219, method=Connection.writePacket, native=false,
column=14, file=app/node_modules/larvitdb/node_modules/mysql2/lib/commands/query.js, function=Query.start, line=43, method=start, native=false,
column=20, file=app/node_modules/larvitdb/node_modules/mysql2/lib/commands/command.js, function=Query.Command.execute, line=38, method=Command.execute, native=false,
column=28, file=app/node_modules/larvitdb/node_modules/mysql2/lib/connection.js, function=PoolConnection.Connection.handlePacket, line=361, method=Connection.handlePacket, native=false,
column=10, file=app/node_modules/larvitdb/node_modules/mysql2/lib/connection.js, function=PoolConnection.Connection.addCommand, line=379, method=Connection.addCommand, native=false
], stack=[
TypeError: value is out of bounds,
at checkInt (buffer.js:825:11),
at Buffer.writeUInt16LE (buffer.js:883:5),
at Packet.writeInt16 (app/node_modules/larvitdb/node_modules/mysql2/lib/packets/packet.js:707:15),
at Packet.writeInt24 (app/node_modules/larvitdb/node_modules/mysql2/lib/packets/packet.js:703:8),
at Packet.writeHeader (app/node_modules/larvitdb/node_modules/mysql2/lib/packets/packet.js:791:8),
at PoolConnection.Connection.writePacket (app/node_modules/larvitdb/node_modules/mysql2/lib/connection.js:219:10),
at Query.start (app/node_modules/larvitdb/node_modules/mysql2/lib/commands/query.js:43:14),
at Query.Command.execute (app/node_modules/larvitdb/node_modules/mysql2/lib/commands/command.js:38:20),
at PoolConnection.Connection.handlePacket (app/node_modules/larvitdb/node_modules/mysql2/lib/connection.js:361:28),
at PoolConnection.Connection.addCommand (app/node_modules/larvitdb/node_modules/mysql2/lib/connection.js:379:10)
]

[sidorares]

Thanks @lillem4n , going to start working on this now

basically we need to support this: https://dev.mysql.com/doc/internals/en/sending-more-than-16mbyte.html

[lillem4n]

@sidorares cool. As I see it this problem is split in two:

1. Async calls should not throw errors, but return them as first parameter of their callbacks. Always.
2. Fix the actual problem. :)

For me this broke the application since it did an unexpected exit.

[lillem4n]

A quick and (kind of) dirty solution is to wrap quite a lot of code in try/catch. I'm laborating a little bit on that now. If you want I can do something that works and make a PR?

[sidorares]

feel free to suggest pr, but I'd rather add extra checks where required (for example, one source of problem here is me trying to store (buffer length >> 16) as byte value - which is out of range for length > 16m). So correct code here should 1) check for 3rd byte to be < 256 2) actually implement the way mysql handles big buffers

On the other hand try/catch around user code ( callbacks ) are necessary

[lillem4n]

Agreed, more sync checks on the right places is needed. Those could then throw errors catched by the try /catch wrapped around to callback gracefully.

I'm going to fiddle around a bit now, then lets see if I fix something good enough for a PR. :)

[sidorares]

mysqljs/mysql related code to compare:

reading - https://github.com/mysqljs/mysql/blob/1aa486e6e0dd6938569949509de6fe66433cc228/lib/protocol/Parser.js#L61-L66

writing - https://github.com/mysqljs/mysql/blob/2aa8b6c8ea2eb93d4d2afa42920362b707e39aed/lib/protocol/PacketWriter.js#L23-L43

mysqljs/mysql#922

[sidorares]

I'm going to fiddle around a bit now, then lets see if I fix something good enough for a PR. :)

Thanks! If you see any kind of unsafe code that input data (valid or malformed) could crash feel free to yell at me @lillem4n

[lillem4n]

While looking around I see I've wrapped all code in a try/catch in my little db layer: https://github.com/larvit/larvitdb/blob/master/larvitdb.js and yet somehow this is circumvented by this error that simply crashes the application.

That means client side try/catch is not enough. I will keep digging until I find a way to catch the errors.

[lillem4n]

Ok, now I have a PR here: #419 that does two things:

1. Adds a test to reproduce the issue (16Mb Buffer INSERT into table)
2. Adds try / catch in the right place so the error is reported back in the .query() callback async instead of thrown sync.

My 2 cents is that this try / catch wrapping and the test should be there even when the actual issue is resolved to more gracefully catch other errors in the future.

Please poke me if I do stuff wrong in your repo with the PR, so I can do better and actually help out next time. :)

[sidorares]

closing this ( big packet support in master, going to npm as 1.1.2 soon ) exception safety tracking issue is #419