Make a v2.0 release?

[wjakob]

The next major version of pybind11 has cooking for quite a while, and now I think it has reached a good balance of brand-new features and general usability improvements.

This ticket is to track what's left to do and make sure that nothing is forgotten. The following items come to mind (please let me know if I missed something):

• py::array flags rework (did not make it for v2.0)
• PyPy support (Python 2.7 branch)
• Make sure that there are no release blockers like bugs involving new features
• Update change log
• Merge & tweak reorganized documentation
• Make a new stable v2.0.0 release.
• Upload to PyPi, Conda, etc.
• anything else?

[aldanor]

First point: not happy, because of the flags. I'll have free time next week and am seriously planning to finish that for good, as discussed in a few other issues here and in my fork :) Now that the few other big parts are already in, like the numpy c api, base/owner support etc, this could be the last big incompatible change for numpy module I think.

[wjakob]

@aldanor: ack, then let's wait for your set of flag changes.

[jagerman]

Perhaps some sort of 2.0.0-RC1 release, to let people sticking with stable know it's coming which might prompt them to try it out before the final release (and thus gain some bug reports for unanticipated problems)?

[dean0x7d]

@jagerman: +1 for RC or some kind of pre-release version.

@wjakob: BTW There's an out-of-date v2.0 milestone over here.

There was a brief discussion about splitting up the Advanced documentation. This may be nice to have ready for v2.0 since the Advanced topics page has become a little frightening for first time visitors. I'm not sure if anyone has already been working on this? If not, I can take a stab at it and reorganize it as previously discussed (1. py::class_ wrappers; 2. py::object API; 3. stl, numpy, eigen casters).

One other thing I wanted to propose was making py::str(arg) do the expected thing, i.e. convert arg into a string. This would deprecate the object::str() member function since having both is a weird gotcha for beginners. Would there be any interest in that?

[jagerman]

Oh, yes, rearranging the docs would be a very nice thing to have for 2.0! I thought about it briefly, but don't really have the time to work on it at the moment. I do think it reached the point a while ago where it's not only daunting for first-time users, but is also not easy for repeat users to quickly find things via the menu. Perhaps moving Advanced to a 3-level organization would work? E.g. keep "Advanced topics" but only have 4-5 general categories within it, with sub-items within those?

[dean0x7d]

Yeah, a little bit of nesting would be good for navigation. I can set up a readthedocs.org fork for a pull request. That way everyone can see the changes properly (in html instead of the raw .rst files) and comment before committing to anything.

[wjakob]

There was a brief discussion about splitting up the Advanced documentation. This may be nice to have ready for v2.0 since the Advanced topics page has become a little frightening for first time visitors. I'm not sure if anyone has already been working on this? If not, I can take a stab at it and reorganize it as previously discussed (1. py::class_ wrappers; 2. py::object API; 3. stl, numpy, eigen casters).

At this point any kind of hierarchical organization will be better than the crazy advanced.rst ;). If you want to take a stab at it, then that sounds all good to me.

One other thing I wanted to propose was making py::str(arg) do the expected thing, i.e. convert arg into a string. This would deprecate the object::str() member function since having both is a weird gotcha for beginners. Would there be any interest in that?

This sounds good to me. One annoying gotcha that might be nice to address at the same time is that std::cout << myObject.str() << std::endl; uses the operator bool() conversion by default unless pybind11/stl.h is included. The fact that this compiles and just writes "1" or "0" can be very confusing and hard to track down.

[wjakob]

@aldanor: any updates on the flags rework? I'd be happy to comment on partial patches.

[aldanor]

@wjakob yep WIP, will keep you posted, the other two seemingly easy PRs took a bit longer than I thought :) The numpy scalars got merged in, but the typed iterators I'll probably finish later, after the flags are done.

[ghisvail]

Any provisional ETA for the v2.0 release folks? Cheers,

[ghisvail]

Right now Debian ships version 1.8.1, but I could push version 2.0 if the latter is to be released quite soon.

[jagerman]

@ghisvail -- I'd expect 2.0 to be out in time for stretch. Speaking as a debian user, it would also be much nicer to have 2.0 in stretch than 1.8.1.

[wjakob]

@ghisvail: That's awesome -- I didn't know pybind11 even had a Debian package ;). By when do you need us to push v2.0 for it to make stretch?

[jagerman]

I didn't know pybind11 even had a Debian package

I didn't either: it's really new (first uploaded to debian unstable ~3 weeks ago).

[jagerman]

@wjakob - the hard freeze for stretch (beyond which packages are only allowed updated versions with justification, e.g. for security updates and similar) is 2016-02-05, but the upload needs to happen at least 10 days before that (a package needs 10 days before transitioning to testing, i.e. stretch). So, basically, the last possible day is around 2016-01-25. (All the gory details on the freeze are at https://lists.debian.org/debian-devel-announce/2016/11/msg00002.html).

[wjakob]

Ok, we'll definitely make that deadline.

[ghisvail]

I didn't know pybind11 even had a Debian package

I should have probably given a heads-up here first. There was not much to say though, the packaging was pretty much effortless for once.

So, basically, the last possible day is around 2016-01-25

I would also need some time to test the update, roll it out and find a sponsor for uploading it. So please consider being (much) more conservative on this deadline.

[wjakob]

I'd like to push it out this year. Already a heads up: there were some changes in the build system due to the merge of #506. In particular, pybind11 now installs .cmake files so that it can be easily autodetected by cmake, and the debian package will need to install those files to the right places. Other than that, it should be pretty much identical (maybe one or two new header files).

[jagerman]

@ghisvail one other change needed for a 2.0 deb: the test target changed to "pytest", which is going to need an override in debian/rules to properly invoke.

[ghisvail]

Why not keep make test instead of make pytest, if I may ask?

[jagerman]

There is a really annoying, long-standing bug in cmake that the test target provided by cmake's built-in ctest support (via enable_testing()) can't have dependencies, which means if you ran make test you wouldn't actually get the necessary rebuild (you have to remember to always run make && make test). CTest's add_test also doesn't add the dependency itself.

As a result, authors stopped using make test in favour of adding a make check target that does testing (even using ctest) but doesn't have the limitation of the severely broken test target that cmake provides on its own.

Now all of that said, we aren't actually using enable_testing() anymore, so I think it actually might be possible to re-add a test target that just depends on pytest.

[jagerman]

Scrap that last bit: even if you aren't under enable_testing(), cmake still reserves the test target and doesn't want it to be used:

  The target name "test" is reserved or not valid for certain CMake features,
  such as generator expressions, and may result in undefined behavior.

[ghisvail]

So, you are essentially trading make && make test for a single make pytest? I am wondering whether it is really worth not adopting the standard just to avoid typing an additional make command. Isn't that something your IDE of choice or a terminal alias could easily handle?

As a result, authors stopped using make test in favour of adding a make check target

When you say "authors", you mean "you guys" right? In my experience dealing with various CMake-based projects, you are the first willing to explicitly bypass ctest for this specific reason. Hence, my curiosity.

FYI, the issue your referred to has been transfered to the new issue tracker. It looks like the interest for it somewhat died down, given the lack of additional discussion.

[dean0x7d]

The extra make && is not the primary reason for the different name. The real issue is what @jagerman mentioned: test is a reserved target name which can only be used with CTest.

CTest is not well suited for running Python tests and we really don't want to reinvent Python testing infrastructure in CTest, hence using off-the-shelf pytest. The only way to use the test name would be to set up CTest (enable_testing(), add_test()) with a single test which calls pytest. This is pretty bad in a lot of ways. The most obvious is that there would be this huge CTest infrastructure as middleware between make test and just a single simple call to pytest. But the real showstopper is that CTest has its own way of handling output which mangles pytest's very nice reporting. Then you have to fight with CTest in the middle... or just call pytest directly and everything's perfect.

[ghisvail]

I see. Thanks for sharing the details.

[jagerman]

make check has become a pretty common alternative, to the point that tools like debhelper will automatically use it if found (and test isn't). It might be worthwhile reintroducing it as a simple target with a dependency on pytest (or even renaming the pytest target back to check). Thoughts?

[dean0x7d]

Yeah, I think it might be good to have check invoke all available test targets. Besides pytest, there are also the CMake function and target tests which live under a separate target. And eventually, embedding support will need its own test (executable instead of a library). check could invoke all three (the individual targets should also remain for selective testing). I'm working on some additional pybind11_add_module functionality and CMake tests now and I'll submit a PR a bit later today, so I think we can move the discussion there.

[wjakob]

I've pushed out a release candidate for v2.0.0: https://pypi.python.org/pypi/pybind11

(One minor annoyance: I called the release v2.0.0-rc1 according to semver, but PyPi changed it to v2.0.0rc1 behind my back)

It's a truly epic release -- typing up the changelog took me the better part of an evening ;). Please let me know in case I missed something: http://pybind11.readthedocs.io/en/master/changelog.html

[ghisvail]

Thanks for the heads-up. Indeed, the changelog is impressive.

[jagerman]

That's quite a list. I can't think of any missing items that I didn't see there.

[ghisvail]

All good for the Debian packaging refresh. It will be pushed soon.

[ghisvail]

@jagerman pybind11/2.0.0~rc1-1 has been uploaded and should be available soon in unstable for you to try on if you like.

@wjakob Should I expect a new RC / stable release within the coming month or should I consider this version to be the one which should eventually land in the next Debian stable?

[wjakob]

@ghisvail: we'll provide a final stable release in the next 1-2 weeks (with minor changes at most). Ideally, that would be the version which lands in debian stable.

[wjakob]

@ghisvail: I just pushed out the final version 2.0.0. It would be great if this lands in Debian (there was a serious issue in the release candidate that is addressed by this new version).

I'm closing this ticket :)