Update to new Arduino library specification

[soundanalogous]

A new library specification will be introduced with Arduino 1.5.3:

https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification

[soundanalogous]

will need to support 2 branches for a while (master and master-1.5.x

[kilianc]

Due doesn't seem to work, after writing F06BF7 on the serial port (CAPABILITY_QUERY) I do not receive any response from the DUE.

P.S. same code works with firmata 2.3 and arduino 2009.

[kilianc]

/cc @rwaldron @divanvisagie

[soundanalogous]

Are you using Arduino 1.5.3?

I just tested a Due and Arduino 1.5.3 (which includes Firmata 2.3.6 in a new directory structure - this is important!) and the firmware.html example for BreakoutJS and I get a capability query response.

All Due users should be using Arduino 1.5.3 or higher from now on. You must also use the version of Firmata that comes installed with Arduino 1.5.3. If you want to use a newer version of Firmata with Arduino 1.5.3 or higher you will have to make sure the library directory structure conforms to the new Arduino 1.5.3+ library style.

[kilianc]

All Due users should be using Arduino 1.5.3 or higher from now on. You must also use the version of Firmata that comes installed with Arduino 1.5.3.

I am!

I changed the code in the serial part of the BreakoutJS server and even there arduino doesn't answer the capability response. May I see your code please?

P.S. examples kinda work, it needs a reboot sometimes :(

/******************** SERIAL ************************/
var serialport = require("serialport");
var serialPort = serialport.SerialPort;
var port = program.port;

var serialDefaults = {
    baudrate: 57600,
    buffersize: 1
};

// Create new serialport pointer
var serial = new serialPort('/dev/cu.usbmodemfa131', serialDefaults);

serial.on("data", function (data) {
    console.log('data', data)

    if (isConnected) {
        connectedSocket.send(data.toJSON());

        if (enableMultiConnect) {
            connectedSocket.broadcast.send(data.toJSON());
        }
    }
});

serial.on("error", function (msg) {
    console.log("serial error: " + msg);
    process.exit(1);
});

serial.on("open", function () {
    console.log('open')
    setTimeout(function () {
        serial.write([0xF0, 0x6B, 0xF7], function (err, results) {
            console.log(err, typeof results)
        })
    }, 5000)
})

[kilianc]

Also this is the test code I am using that runs on the 2009 and not on DUE

May I see your output please :D

var usbPath = '/dev/cu.usbmodemfa131' // change it 
// var usbPath = '/dev/tty.usbserial-A9007LAv'
var SerialPort = require('serialport').SerialPort
var serialPort = new SerialPort(usbPath, { baudrate: 57600, buffersize: 1 })

serialPort.open(function () {
  console.log('open')

  serialPort.on('data', function (data) {
    console.log('data received: ' + data)
  })

  serialPort.on('error', function (err) {
    console.log('error event:', err)
  })

  setTimeout(function () {
    console.log('writing...')
    serialPort.write(new Buffer('F06BF7', 'hex'), function (err, results) {
    // serialPort.write([0xF0, 0x6B, 0xF7], function (err, results) {
      console.log(err, typeof results, results)
    })
  }, 5000)
})

[soundanalogous]

Had some time to look into this further. There is a bug here indeed. I'm not sure yet if the bug is with the Due or Firmata (but I'm suspecting it may be a Due issue). With the Due it seems that the first TX sent after the serial port connection is made will be ignored (either that or the first TX from the Due is dropped). You can get around it by sending the report firmware version command (see updated code below). Since that is the first TX, it will be ignored and all subsequent transmissions will work. I found that this first TX also has to be a command that would normally return a response. For example I tried to send 0xFF (system reset), but that did not fix the issue (possibly because it is a command that does not return a response). I'll look into this further. If it turns out to be a Due bug there is not much I can do other than report it to the Arduino team. I'll see if there is anything I can do in Firmata to get around the Due bug (such as a serial line flush).

I didn't notice this bug with BreakoutJS because with breakout server (either version) you only connect to the serial port once. After than any browser refresh will perform a soft reset so that would get around the Due issue. Guess I need to test with more Firmata client libraries more often.

var usbPath = '/dev/cu.usbmodemfd121' // change it 
// var usbPath = '/dev/tty.usbserial-A9007LAv'
var SerialPort = require('serialport').SerialPort;
var serialPort = new SerialPort(usbPath, { baudrate: 57600, buffersize: 1 });

function write (buffer) {
  serialPort.write(buffer, function (err, results) {
    console.log(err, typeof results, results)
  })
}

serialPort.open(function () {
  console.log('open')

  serialPort.on('data', function (data) {
    // you'll need toJSON() to see the data
    console.log('data received: ' + data.toJSON())
  })

  serialPort.on('error', function (err) {
    console.log('error event:', err)
  })

  setTimeout(function () {
    console.log('writing...')

    // bug: the first write after the serial port connection is made does
    // not return a response. Increasing the delay time at startup does not
    // make a difference either. It appears that this issue is unique to the
    // Due.

    // hack: send a report firmware version request. It will fail but 
    // subsequent writes will work. Issue here is that with other boards this 
    // will not fail and you'll get a report firmware response.
    write(new Buffer('F079F7', 'hex'));

    // subsequent writes work
    write(new Buffer('F06BF7', 'hex'));

  }, 2000)
})

[soundanalogous]

Turns out there is definitely garbage in the Due serial input stream just after a serial connection is made. The issue I suspect is described here: arduino/Arduino#1267. It requires a firmware update of the ATMega16U2 on the Due board - this is the microcontroller that handles the usb to serial conversion.

However, I did manage to figure out a software fix for this for Firmata. Adding the following just after line 73 in Firmata.cpp will fix the issue for the Due:

// clear any garbage in the input stream (this fixes an issue with Due)
while (FirmataSerial->read() != -1);

I'm not sure yet if this causes issues with any Firmata client libraries since any incoming serial data sent during the blink version routine will be lost - users shouldn't try to send data during that time anyway). I tried including that line of code before the blinkVersion routine, but it did not work. It seems the garbage gets in the input stream some number of micro or milliseconds after the Serial.begin method is called (which further supports the ATMega16U2 bug theory).

[kilianc]

@soundanalogous I confirm that your fix works.

Also other people can download the patched firmata.cpp here
I am going to update the ATMega16U2 firmware tomorrow since I would probably blow up the whole block now 💤

Thanks @soundanalogous!

/cc @rwaldron @divanvisagie

[soundanalogous]

I updated the ATMega16U2 firmware and it did not fix the issue. The Firmata patch will have to work for now. The line should also be added after the call to printFirmwareVersion in the 3rd override of the begin method in case anyone does something like this in StandardFirmata (or similar) to bypass the startup blink sequence:

Serial.begin(57600);
Firmata.begin(Serial);
// you can also use other hardware serial ports as well as Ethernet and Wi-Fi streams this way.

update - a delay is also necessary when overriding the call to Firmata.begin:

/* begin method for overriding default stream */
void FirmataClass::begin(Stream &s)
{
  FirmataSerial = &s;
  // do not call blinkVersion() here because some hardware such as the
  // Ethernet shield use pin 13
  delay(800); // required for issue with Due
  printVersion();
  printFirmwareVersion();
  // clear any garbage in the input stream (this fixes an issue with Due)
  while (FirmataSerial->read() != -1);
}

Something really strange is happening. I'm suspecting the ATMega16U2 may be resetting after the serial connection is made and then it's getting garbage characters on the input stream after reset. I'll bring this up with the Arduino team.

[ntruchsess]

I noticed that the configurable-branch doesn't compile on 1.5.4.
Just putting eveything in Firmata-lib dir as for 1.0.x IDE makes the library disappear from example-directory.
Putting Firmata.h, Firmata.cpp, Boards.h and utility in src, leaving library.properties, keywords.txt and examples top-level makes the examples re-appear, but results in compile-errors no matter which example you try (because it allways tries to compile the files in utility). It seems to be caused by using the path 'utility/' in the sketch, but not in die Feature-classes itself).
Putting the files from utility into arch/default or arch/defaut/utility also both does not work but results in compile-errors (either includes, or type not found - depending whether the files are in a utility-subdir or not).

[soundanalogous]

I've added the shell script (release.sh) that I've been using to create a packaged version of Firmata. It creates a zip for Arduino 1.0.x and 1.5.x. Haven't tested with 1.5.4 but I did not encounter any issues with 1.5.3. I created this script specifically for OS X, but the only OS X specific part is the deletion of .DS_Store files of it may work on Linux and Win (cygwin) as well without modification. Feel free to modify to support other platforms if necessary.

Also, there is a large discussion on the Arduino developers list regarding the Arduino 1.5.x library structure. I'm suspecting things could change before the next Arduino release.