python 3 not supported

[pwolfram]

At present, MPAS-Analysis is not working on python 3.5. This issue is meant to be a dialog about which versions of python and which packages we wish to support. See also #40, #42

Partial list of needed changes:

• change from % formatting of string to .format(...)

[pwolfram]

cc @xylar, @milenaveneziani, and @mark-petersen

[xylar]

@pwolfram, I think this it is a good idea to support python 3.5 and I think this is the repo (along with geometric_features) to use to start doing that. That being said, I've never written a piece of python 3.5 code in my life so it'll be a learning experience for me. But I think it's foolish to write code for python 2.7 just because it's familiar and because (currently) more libraries seem to support it.

[pwolfram]

@xylar, python 3.5 is very similar with some new and depreciated features. In principle, it shouldn't be too difficult to support both for now and eventually depreciate use of python 2.7 in the future.

[xylar]

As I go through an make files PEP8 compliant, I've also been changing from % formatting of string to .format(...), which seems to be a requirement for supporting python 3.x. Others should try to move to this approach as well.

If there are other habits we need to change, it wouldn't hurt to identify them earlier rather than later.

[xylar]

@jhkennedy, we've got an issue here to remind ourselves to support python 3, but we haven't actually tried very hard. If you could suggest some habits we need to break to support python 3, that would be helpful. Otherwise, I suspect it would just be a matter of testing the code and seeing what breaks.

One issue I've seen that might cause python 3 issues is iterating through dictionaries (with items() vs. iterateitems()). I've been using items() on the understanding that it's more python 3 friendly...

Also, something or other came up in #240 (generating the website for results) where I wanted to get the first item in an ordered dictionary and the syntax was markedly different for python 2 and 3, and I went with the python 2 syntax for now...

[jhkennedy]

Overall, writing python 2.7 and 3 compatible code, requires very little stylistic changes (supporting python 2.5 and earlier gets more complicated; 2.6 is mostly okay, with some gotchas).

Python 3 has some good documentation on porting python 2 code, but I actually found going through the python-future cheat sheet much easier to digest.

This blog post illustrates the changes pretty well as well.

In terms of writing the code, there are four changes that are most apparent: print becomes a function, relative imports need to be declared as such, division becomes non-truncating by default, and strings are unicode by default instead of bytes. Quick examples:

print 'hello'  # py2
>>> hello
print('hello') # py3
>>> hello

import cvs        # py2: local package hides stdlib package; py3: stdlib package loaded
from . import cvs # py2.5+ and 3: local package hides stdlib packge

5 / 4  # py2
>>> 1

5 / 4  # py3
>>> 1.5

5 // 4 # py2.2+ and 3
>>> 1

By placing these imports:

from __future__ import absolute_import, division, print_function, unicode_literals

at the very top of the python files, python 2 will behave like python 3 in these instances (and 3 will still be 3).

Note: unicode_literals should be used with caution as it my change your API by returning unicode instead of the previous 8-bit strings.

Also, some modules in the stdlib have been renamed, or combined in python 3, so the package six (six stands for python 2*3) provides a handy way to import the packages using the python 3 stdlib name, as well as some of the particularly useful python 3 functionality that isn't back ported to python 2 (e.g., reraise() for exceptions).

---

Iterating does change a bit in how it functions. In python 3, many things that returned as a list now return an iteration-generator. So, in python 2, dict.items() actually constructed a list of (key, val) tuples, and returned that. This causes a bit of extra memory and process overhead, so it was recommended to use dict.iteritems() to get an iteration-generator. Python 3, decided only have the iteration-generator style as a class method, so now dict.items() will return an iteration-generator (if you want the list, you just wrap it in a list()).

So, if you don't care about the performance hit in python 2, you can just use dict.items() in the code. If you do care, six provides the six.iteritems() function which will do what's fastest for the python version used.

---

As for the first item of an OrderedDict, a python 2.6+ and 3 compatible way would be to use next(iter(OrderedDict)) pattern, which you can test with this little example across versions:

from collections import OrderedDict
d = OrderedDict( [('a', 0), ('b', 1), ('c', 2)] )
next(iter(d))

[xylar]

Thanks @jhkennedy, these are some great tips. I'm happy to learn that next(iter(OrderedDict)) is an option in any python version we would want to support. Quite an easy fix to one problem at least...

As to using six, my preference would be to avoid it where possible, since performance in things like looping through dictionaries is definitely not an issue for us.

The integer division one is an interesting case and one of the harder cases to search for. I don't think we're doing much (if any) of that but perhaps my c++ background got the better of me once or twice?

Anyway, fun stuff ahead adding support for python 3! And I'm enough of a nerd that I'm not even being sarcastic!

[xylar]

@milenaveneziani (cc @jhkennedy, @pwolfram), I just had a discussion with the workflow team. acme_diags and UV-CDAT are rapidly adding python 3 support. If I understood right, this was strongly pushed for in a recent ESG review.

Anyway, the e3sm-unified environment would be a lot simpler to maintain if we were to migrate everything to supporting both python 2 and 3, and to switch to python 3 as our main working environment. This means that this issue, which has been on the back burner, now has some urgency. Ideally, we would have python 3 support before v1.0 gets released.

This would also presumably affect a-prime, which would also need to fully support python 3.