Release the GIL

[mrocklin]

Would it be reasonable for bottleneck to release the GIL in critical sections of Cython code?

cc @jreback @shoyer

[kwgoodman]

I think it is a good idea. I don't know of any downsides (can anyone think of any?).

[shoyer]

This is definitely a good idea, and in fact I added it in many places to the previous version of bottleneck. But it looks like it got lost in the v1.0 rewrite.

[mrocklin]

The two main costs are probably

1. Human hours and finding someone to do it
2. The possibility that bottleneck implicitly relies on the GIL (unlikely in this case I think)

There has been some recent interest in this lately, namely scikit-image/scikit-image#1519 and pandas-dev/pandas#10199

[shoyer]

@mrocklin Is it possible to use the NumPy C API with the GIL released? The answer to that question will tell us whether or not this is viable for most of bottleneck.

[seibert]

Unfortunately, I think the answer is "mostly yes", but not "always yes". There are some bits of the Numpy C API that indicate they require the GIL because they manipulate Python reference counts. The two examples I found said there would be problems related to Numpy arrays with an object dtype, and some C API calls relating to copying iterators.

[shoyer]

@seibert OK, I think we should be mostly fine for bottleneck, then. Thanks!

[seibert]

A warning from Travis: "Only some specific parts of the NumPy C API can be used with the GIL released. Most of the C-API will have some reference counting code and therefore require the GIL."

[kwgoodman]

I released the GIL in all bottleneck functions. It's in master. Give it a try and let me know how it goes.

[kwgoodman]

No interest in testing bottleneck's releasing of the GIL? Oh, well. I'll close this as I have opened #118 on running bottleneck with multiple threads.

[mrocklin]

My apologies, this slipped through my e-mail.