Feature: On-target execution tests [WIP] #147

agnat · 2015-07-10T22:15:28Z

Still work in progress, but fun to play with. From the README:

AVR LLVM Integrated Tester

This tool builds an AVR executable from test code lowered by our backend and the
libavrlit test suite library using a known good toolchain (avr-gcc). The
resulting binary is uploaded to a development board. Test results are collected
using a virtual tty.

Setup

Things you will need:

ATmega32U4 board with USB and AVR-109 type bootloader. (Arduino Leonardo,
RedBear Blend (Micro), &c.)
pySerial python module
Fire extinguisher

Set the AVRLIT_PORT environment variable to the tty path of your board.

> export AVRLIT_PORT=/dev/tty.usbmodemfd141

If your board currently runs an Arduino sketch that uses the serial port, you are
all set. Otherwise, you need to reset the board manually for the first run.

> bin/llvm-lit -v ../llvm/test/CodeGen/AVR/

Writing Tests

The on-target execution tests reside in llvm/test/CodeGen/AVR. Like other lit
tests they contain a RUN: line calling llvm-avrlit:

// RUN: llvm-avrlit %s %p/add.ll 

#include <avrlit.h>

using namespace avrlit;

extern "C" {  // actually this is extern "IR" but Bjarne forgot.
  i8 add8_reg_reg(i8, i8);
}

AVRLIT_TEST(llvm_avr) {
  reenter (this) {  // don't worry about the coroutine
    plan(3);
    ok(_(add8_reg_reg( 23, 42) ==  23 + 42)); yield;
    ok(_(add8_reg_reg(-23, 42) == -23 + 42)); yield;
    ok(_(add8_reg_reg(-23, 42) == 0));        yield;
  }
}

All of this is still in flux. I'll explain it if I decide to keep it. ;)

agnat · 2015-07-11T02:16:10Z

Ready for review... and test drive ;)

dylanmckay · 2015-07-13T01:18:29Z

Where is add8_reg_reg &co defined?

Also, do you think this would be worth putting into a seperate repository? It is quite a bit less lightweight than any other test, and won't be useful to users.

agnat · 2015-07-13T13:23:02Z

add8_reg_reg &co are defined in add.ll &co.

Take a look at the // RUN: ... line above and $OBJ/test/CodeGen/AVR/execute_on_target_test/Makefile. Yeah... I know ... totally cool... :-D

I'd rather keep it in-tree. It's only a hand full of files and we're talking about the LLVM repository. It's huge no matter what. Let's drown out the static overhead by writing a bazillion tests.

In the long run I'd like to make it part of the CI. All we need is a build bot and an Arduino.

agnat · 2015-07-13T16:13:07Z

Ah, right. you don't have a board yet. Here is the makefile...

# Generated by llvm-avrlit. Do not edit.
AVRLIT_BOARD  ?= leonardo
AVRLITD = /Users/david/projects/avr-llvm/llvm/utils/avr/avrlit/libavrlit

include $(AVRLITD)/$(AVRLIT_BOARD)/board.mk

execute_on_target_test.$(AVRLIT_BOARD).hex:

clean:
    rm -f *.hex *.elf *.o

.PHONY: clean

execute_on_target_test.$(AVRLIT_BOARD).hex : execute_on_target_test.$(AVRLIT_BOARD).elf
    avr-objcopy -O ihex -R .eeprom -R .fuse -R .lock -R .signature $< $@

execute_on_target_test.$(AVRLIT_BOARD).o : /Users/david/projects/avr-llvm/llvm/test/CodeGen/AVR/execute_on_target_test.cpp
    avr-g++ -Os -mmcu=$(MCU) -std=c++11 -I$(AVRLITD) -c $< -ffunction-sections -o $@

add.$(AVRLIT_BOARD).o : /Users/david/projects/avr-llvm/llvm/test/CodeGen/AVR/add.ll
    llc -mtriple=avr-atmel-none $< -filetype=obj -o $@

and.$(AVRLIT_BOARD).o : /Users/david/projects/avr-llvm/llvm/test/CodeGen/AVR/and.ll
    llc -mtriple=avr-atmel-none $< -filetype=obj -o $@

xor.$(AVRLIT_BOARD).o : /Users/david/projects/avr-llvm/llvm/test/CodeGen/AVR/xor.ll
    llc -mtriple=avr-atmel-none $< -filetype=obj -o $@

execute_on_target_test.$(AVRLIT_BOARD).elf: execute_on_target_test.$(AVRLIT_BOARD).o add.$(AVRLIT_BOARD).o and.$(AVRLIT_BOARD).o xor.$(AVRLIT_BOARD).o
    avr-g++ -Os -mmcu=$(MCU) -std=c++11 $^ -L$(AVRLITD)/$(AVRLIT_BOARD) -lavrlit -Wl,--gc-sections -Wl,--relax -o $@

Also note that usb.h, usb.c and LUFAConfig.h are pretty boilerplate. The interesting things happen in avrlit.h, avrlit.cpp and llvm-avrlit.in.

agnat · 2015-07-14T14:18:58Z

I found this post regarding the PIC backend a while back, lost it, and found it again.

Quote:

However, a back-end maintainer has to provide a few "guarantees" to continue supporting in tree:

You must have a public buildbot that can test your code. Since you'll always cross compile, having a good set of tests in x86 mode would do, but having execution tests would increase the quality by a large amount. If at all possible, I'd recommend a cross-compilation buildbot that runs some embedded code in a proper PIC.

agnat · 2015-07-14T14:28:31Z

One defect of the current design is that it allows only one target execution test. That's because llvm-lit runs tests in parallel. Since llvm-avrlit does not perform any locking there currently is a race on the tty device (chaos).

We have a little time on this. In the long run we need some sort of serialization.

dylanmckay · 2015-07-16T01:29:17Z

Can you add e0cc7d5, I can't seem to push a commit to your PR (probably doing it wrong).

Note that I must run llvm-lit as root otherwise the script is unable to access the tty device.

When I do, llvm-lit succeeds - but the on_target_execution_test.cpp test is marked unsupported.

agnat · 2015-07-16T01:50:25Z

Can you add e0cc7d5 [...]

Done.

Note that I must run llvm-lit as root otherwise the script is unable to access the tty device.

Hm, not good. There probably is a cleaner (distro specific) way to do this. There often it is a user group you need to member in to access the ttys.

When I do, llvm-lit succeeds - but the on_target_execution_test.cpp test is marked unsupported.

Unsupported means AVRLIT_PORT is not set. This is handled in lit.local.cfg. You probably use sudo?... which creates a fresh environment loosing your setting in the outer shell? I'm guessing...

dylanmckay · 2015-07-16T09:52:02Z

Could you also add 8982a31.

I got it to run by adding my user to the relevant groups, however llvm-lit hangs when running the test. I am not sure if this has something to do with the board I'm using - a Freetronics LeoStick. It is a leonardo compatible however.

Feel free to merge,

agnat · 2015-07-16T13:17:54Z

Pretty sure it is not the board. I'd guess it's either a timing issue (I tested it on OS X only) or more python incompetence on my end.

If you don't feel like debugging it just drop your distro name. Then I'll set up a vbox.

If you want to look into it, here are a few things to try:

Try running llvm-avrlit ... manually to get console output. You'll need to add $OBJ/bin to your path, so it does find our llc.
Try this line as a standalone script. It should throw your stick into the boot loader. If it doesn't, add a bit of delay before closing the tty. (These boards reset when the tty is closed while being set to 1200bps).
Upload $OBJ/test/CodeGen/AVR/execute_on_target_test/execute_on_target_test.leonardo.hex manually using avrdude
Look at the test output using your favorite terminal (CoolTerm, minicom, screen)
print(...)-debug the living 💩 out of it...

Could you also add 8982a31.

Done.

Feel free to merge,

Will do.

This adds two things: libavrlit: A small testing framework for the AVR, built using avr-gcc. It reports results to a TTY using a format similar to llvm-lit. llvm-avrlit: A test driver script. It builds an AVR executable from code lowered by our backend and libavrlit. It uploads and runs the test executable on a test board (Arduino Leonardo et al.) and collects the results. The on-target tests are only enabled if the AVRLIT_PORT environment variable is set. This is checked in test/CodeGen/AVR/lit.local.cfg.

dylanmckay · 2015-07-16T13:39:19Z

I'll follow your steps in the next few days, I'll update once I get around to it :)

[AVR] Add on-target execution tests

agnat force-pushed the feature/on_target_execution_tests branch from 5ec3ef1 to 3157b71 Compare July 11, 2015 02:00

agnat force-pushed the feature/on_target_execution_tests branch from 8982a31 to 8464221 Compare July 16, 2015 14:32

agnat added a commit that referenced this pull request Jul 16, 2015

Merge pull request #147 from avr-llvm/feature/on_target_execution_tests

e2a9559

[AVR] Add on-target execution tests

agnat merged commit e2a9559 into avr-support Jul 16, 2015

agnat deleted the feature/on_target_execution_tests branch July 16, 2015 15:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature: On-target execution tests [WIP] #147

Feature: On-target execution tests [WIP] #147

Uh oh!

agnat commented Jul 10, 2015

Uh oh!

agnat commented Jul 11, 2015

Uh oh!

dylanmckay commented Jul 13, 2015

Uh oh!

agnat commented Jul 13, 2015

Uh oh!

agnat commented Jul 13, 2015

Uh oh!

agnat commented Jul 14, 2015

Uh oh!

agnat commented Jul 14, 2015

Uh oh!

dylanmckay commented Jul 16, 2015

Uh oh!

agnat commented Jul 16, 2015

Uh oh!

dylanmckay commented Jul 16, 2015

Uh oh!

agnat commented Jul 16, 2015

Uh oh!

dylanmckay commented Jul 16, 2015

Uh oh!

Uh oh!

Feature: On-target execution tests [WIP] #147

Feature: On-target execution tests [WIP] #147

Uh oh!

Conversation

agnat commented Jul 10, 2015

AVR LLVM Integrated Tester

Setup

Writing Tests

Uh oh!

agnat commented Jul 11, 2015

Uh oh!

dylanmckay commented Jul 13, 2015

Uh oh!

agnat commented Jul 13, 2015

Uh oh!

agnat commented Jul 13, 2015

Uh oh!

agnat commented Jul 14, 2015

Uh oh!

agnat commented Jul 14, 2015

Uh oh!

dylanmckay commented Jul 16, 2015

Uh oh!

agnat commented Jul 16, 2015

Uh oh!

dylanmckay commented Jul 16, 2015

Uh oh!

agnat commented Jul 16, 2015

Uh oh!

dylanmckay commented Jul 16, 2015

Uh oh!

Uh oh!