gh-82012: Deprecate bitwise inversion (~) of bool

[timhoffm]

The bitwise inversion operator on bool returns the bitwise inversion of the underlying int value; i.e. ~True == -2 such that bool(~True) == True.

It's a common pitfall that users mistake ~ as negation operator and actually want not. Supporting ~ is an artifact of bool inheriting from int. Since there is no real use-case for the current behavior, let's deprecate ~ on bool and later raise an error. This removes a potential source errors for users.

Full reasoning: #82012 (comment)

• Issue: bool(~True) == True #82012

[hugovk]

Please document this in What's New at https://docs.python.org/3.12/whatsnew/3.12.html#deprecated

And is this operator documented in the main reference? Let's also mention the deprecation there.

Is the plan to turn this into an error in 3.14, or keep it open ended?

[gvanrossum]

LGTM. Let's just merge this.

[iritkatriel]

We should probably add a comment about this to https://github.com/python/cpython/blob/main/Doc/whatsnew/3.12.rst.

[gvanrossum]

We should probably add a comment about this to https://github.com/python/cpython/blob/main/Doc/whatsnew/3.12.rst.

Yes!

[timhoffm]

I've added the deprecation notice to whatsnew.

Is the plan to turn this into an error in 3.14, or keep it open ended?

I suggest to turn this into an error (and have indicated so in the whatsnew). There is a real danger that some users write if ~condition (which is always True). Warnings are often overlooked, so it's better to eventually error on this. In contrast, I can hardly imagine any intended use for ~some_bool, so this error should not annoy any rightful users.

[JelleZijlstra]

We should test this behavior separately (doing something like with assertWarns(DeprecationWarning): exec("~False")).

[timhoffm]

Do you mean instead of or in addition to the current test? We originally had eval("~True") but I figured it's clearer to have the warning context only around the operation and not around a comparably complex eval() or exec(). I'm happy to change if there's a benefit of these though.

[JelleZijlstra]

We should have both. The existing tests check that the deprecation warning is emitted correctly at runtime. The new tests would ensure that if the operation occurs at compile time, we still emit the DeprecationWarning.

[JelleZijlstra]

The new behavior should also be mentioned in https://docs.python.org/3.12/library/stdtypes.html#boolean-values

It currently says "In numeric contexts (for example when used as the argument to an arithmetic operator), they behave like the integers 0 and 1, respectively.", which will no longer be true in all contexts.

[timhoffm]

The new behavior should also be mentioned in https://docs.python.org/3.12/library/stdtypes.html#boolean-values

It currently says "In numeric contexts (for example when used as the argument to an arithmetic operator), they behave like the integers 0 and 1, respectively.", which will no longer be true in all contexts.

I suggest to not document this as it would become more confusing than helpful:

In my view, there should be no reason to use bitwise operators on bool; one should always use the boolean operators (or convert to int explicitly). I therefore consider all bitwise operators of bool an implementation detail. While this is public API, it's not something we should advertize.

In particular, if we document ~, we'd also need to leave some words on &, | and ^. And then it gets quite messy. If they operate on bool, they return bool. In contrast, ~ was returning int, will now warn and will error out in the future. Changing to returning bool for ~ meaning a logic negation was rejected in the original issue discussion as too much of an API break (still working but with changed behavior). If I had to document this correctly this would be:

The binary bitwise operators &, | and ^ on two bools return a bool and behave like their logical equivalents. When applying these binary bitwise operators to mixed bool and int arguments, they return an int and interpret the bool as the underlying int (i.e. 0 or 1). The bitwise negation ~ of a bool currently returns the bitwise complement of the underlying int. This is deprecated and will raise an error in 3.14. Generally, the use of bitwise operators on bools is discouraged. Use the logical operators and, or, not and xor instead, or convert to int explicitly.

But maybe you have better ideas what to document.

[JelleZijlstra]

The new behavior should also be mentioned in https://docs.python.org/3.12/library/stdtypes.html#boolean-values
It currently says "In numeric contexts (for example when used as the argument to an arithmetic operator), they behave like the integers 0 and 1, respectively.", which will no longer be true in all contexts.

I suggest to not document this as it would become more confusing than helpful:

In my view, there should be no reason to use bitwise operators on bool; one should always use the boolean operators (or convert to int explicitly). I therefore consider all bitwise operators of bool an implementation detail. While this is public API, it's not something we should advertize.

Whether we like it or not, these operators have worked this way for a long time in Python, and backwards compatibility alone means we can't just make them go away. So it's better to document clearly how they work and what is changing.

In particular, if we document ~, we'd also need to leave some words on &, | and ^. And then it gets quite messy. If they operate on bool, they return bool. In contrast, ~ was returning int, will now warn and will error out in the future. Changing to returning bool for ~ meaning a logic negation was rejected in the original issue discussion as too much of an API break (still working but with changed behavior).

I think the whole reason that we're deprecating only the ~ operator is that it's unlike the others: the results of &, |, and ^ make sense whether you think of bools or ints, it's only ~ that turns something from a bool into a non-bool int.

If I had to document this correctly this would be:

The binary bitwise operators &, | and ^ on two bools return a bool and behave like their logical equivalents. When applying these binary bitwise operators to mixed bool and int arguments, they return an int and interpret the bool as the underlying int (i.e. 0 or 1). The bitwise negation ~ of a bool currently returns the bitwise complement of the underlying int. This is deprecated and will raise an error in 3.14. Generally, the use of bitwise operators on bools is discouraged. Use the logical operators and, or, not and xor instead, or convert to int explicitly.

But maybe you have better ideas what to document.

Your wording sounds good to me!

[timhoffm]

@JelleZijlstra thanks for the feedback. I've rewritten and restructured the bool docs a bit so that it's hopefully more clear and still precise. Please check if this is ok. - It's in a seprate commit, so could be easily modified/reverted.

In particular, I've created a top-level section "Boolean Type - bool", which does the importance of the type more justice.
Before, the bool description was only in documented in the section "Other Built-in Types / Boolean Values". I've moved most of the description over from there, but left the section as a short stub; both because (1) I'm unclear whether we still need "Boolean Values" explicitly in addtion to "Boolean Type", and (2) because that keeps the link so that internal and external references are not broken..

[JelleZijlstra]

Looks good, but I'd like to have another core dev look at the docs before we merge.

[hauntsaninja]

I think I'd prefer to remove the Boolean Values section. We can preserve the .. _bltin-boolean-values: and index roles, which should hopefully preserve links.

[timhoffm]

I think I'd prefer to remove the Boolean Values section. We can preserve the .. _bltin-boolean-values: and index roles, which should hopefully preserve links.

You mean moving them to the new "Boolean Type" section?

[hauntsaninja]

Yup!

[timhoffm]

I think I'd prefer to remove the Boolean Values section. We can preserve the .. _bltin-boolean-values: and index roles, which should hopefully preserve links.

HTML links to the section cannot be preseved the URL was https://docs.python.org/3.12/library/stdtypes.html#boolean-values and the anchor #boolean-values is constructed from the section title, not from the label.

I've rewritten internal references (because the label _booltype is more in line with the labeling conventation in the file than _bltin-boolean-values). And the moved index roll works.

[gvanrossum]

Is this waiting for anyone?

[hauntsaninja]

Not anymore. Thanks all!

[pllim]

Hello. The removal of this ref tag broke intersphinx downstream. I have not tracked down yet which middleware is calling this tag when API docstring has a "bool" mentioned. I suspect it is numpydoc. Is it not possible to reuse this tag for your new section above?

xref astropy/astropy#15428

[hauntsaninja]

If you open a PR, I can backport it to the 3.12 branch

[pllim]

Sure. Please see #110371 . Thank you for your consideration!

[josyb]

IMHO deprecating the ~ operator while keeping &, | and ^ seems (very?) inconsistent to me.
As bool is a subclass of int one could expect that ~bool(0) returns True, sadly it returns -1 ...
As a Hardware guy I see the bool as a constrained int with a bit width of 1, with 0 being equivalent to False and 1 to True, so the negation ~ would simply flip the bit. Especially as e.g. int(bool(1e6)) returns 1
With this you are breaking a lot of MyHDL code and probably a lot of others too.
I guess if you fix ~bool() less code will get broken, possibly none at all. I can hardly imagine anyone relying on ~bool(0) returning -1 and ~bool(1) returning -2

[timhoffm]

As bool is a subclass of int one could expect that ~bool(0) returns True, sadly it returns -1 ... As a Hardware guy I see the bool as a constrained int with a bit width of 1, with 0 being equivalent to False and 1 to True, so the negation ~ would simply flip the bit. Especially as e.g. int(bool(1e6)) returns 1

Unfortunately, this is not how bool was implemented. It's a full integer underneith. This means that be bit inversion of True (0000 ... 0001) is 1111 ... 1110. This in particular means that bit inversion does not behave as a logical negation: bool(~True) is True. This is a major pitfall and the motivation for the deprecation. To be clear: If you have used ~ as a logical negation, your code is broken.

IMHO deprecating the ~ operator while keeping &, | and ^ seems (very?) inconsistent to me.

The other bitwise operations happen to behave like the logical variants, e.g. bool(a & b) == a and b. So, while not recommended (The logical and operator is and, not &), there's not the danger of getting unexpected results. To keep changes minimal, we've kept these operators.

With this you are breaking a lot of MyHDL code and probably a lot of others too.

As above, if you have used ~ as logical negation, the existing code does not do what you think. The deprecation should make you aware of this. Please switch to the logical negation not.

[zhijieshi]

It is not quite true that "If you have used ~ as a logical negation, your code is broken."

Using bitwise operations has some advantages over logical operations, e.g., in the context of MyHDL package, which is for hardware design. Not matter how many bits are in signals, we can write the same expression. We just need to mask out unnecessary bits in the end.

output_1bit = (a & ~b) & 1
output_4bits = (x & ~y) & 0xF

Edit: I did more testing. It seems the above code actually works. I only saw warnings on (single) bit indexing.

[timhoffm]

Using bitwise operations have some advantages over logical operations

Agreed. But to have and advantage of bitwise, your data should have multiple bits, i.e. the underlying data type should be int not bool. Otherwise you're implicitly casting from bool to int somewhere in your expressions. In general I would claim: If you have bool data, use logical operators, if you have int data, you can use the bitwise operators. And do any typecasts explicitly.

In that sense, inputs to your examples should be ints.

Remark: While I agree that it would be nice if ~b would return the negated True/False on a bool, we decided against that for safety. Unfortunately, the current behavior is different and switching to another working behavior was considered too dangerous, see the original discussion in (#82012).

[josyb]

I have been unavailable this Sunday, so I will respond chronologically

@timhoffm
#103487 (comment):

Unfortunately, this is not how bool was implemented. It's a full integer underneith.

So it is basically flawed? (rhetorical question ...) A bool that nonetheless pretends to be an int?

The other bitwise operations happen to behave like the logical variants, e.g. bool(a & b) == a and b

Oh, we can get away with this ...
Actually CPython takes care to return a bool - see the code later

In MyHDL (which is 100% Python) this:

    a = Signal(intbv(0)[2:])

    y.next = (~a[1] & a[0]) | (a[1] & ~a[0])

a holds a constrained unsigned integer with bit width 2; indexing a returns a bool
y holds a bool

Above code simulates (== running the actual code as it is Python) returning correct results - so it is not broken, but we will have to fix it ...

For the record: above codes comes from @zhijieshi; I always use not, and, or; as in VHDL , but in Verilog it is ~, &, | ; having both styles in Python can be / is confusing; but IMNSHO they should act the same.

@zhijieshi
#103487 (comment)
Your final & 1 is not always necessary, but it is better to be safe than sorry.
It will have a time penalty in the simulation; the conversion to either Verilog or VHDL will also have this superfluous final and but the synthesizer will remove that.
Yes, the only issue is the singular case as:

   y.next = ~r

But MyHDL will throw a ValueError as -1 nor -2 will fit into the destination; be it an intbv(0)[1:] or a bool()

@timhoffm
#103487 (comment)

But to have and advantage of bitwise, your data should have multiple bits, i.e. the underlying data type should be int not bool

You are defending the choice of deprecating the ~ operator on bool instead of fixing it.
@gvanrossum words:
#82012 (comment)

Probably when we introduced bool we should have thought harder about it, but I don't think we should change anything at this point, so I'm not sure why whether it's worth trying to uncover the original deep motivations (probably they weren't so deep).

With bool being a subclass of int we can expect that bool to follow the int rules. You say that there is an unsafe case because of the actual implementation of bool and there is. I am quite sure that the followers of ~ will have encountered that; in fact @zhijieshi has, so he added the and 1 to protect himself.

I took a look into the GitHub repo:

/* Arithmetic operations redefined to return bool if both args are bool. */

static PyObject *
bool_invert(PyObject *v)
{
    if (PyErr_WarnEx(PyExc_DeprecationWarning,
                     "Bitwise inversion '~' on bool is deprecated. This "
                     "returns the bitwise inversion of the underlying int "
                     "object and is usually not what you expect from negating "
                     "a bool. Use the 'not' operator for boolean negation or "
                     "~int(x) if you really want the bitwise inversion of the "
                     "underlying int.",
                     1) < 0) {
        return NULL;
    }
    return PyLong_Type.tp_as_number->nb_invert(v);
}

static PyObject *
bool_and(PyObject *a, PyObject *b)
{
    if (!PyBool_Check(a) || !PyBool_Check(b))
        return PyLong_Type.tp_as_number->nb_and(a, b);
    return PyBool_FromLong((a == Py_True) & (b == Py_True));
}


static PyObject *
bool_or(PyObject *a, PyObject *b)
{
    if (!PyBool_Check(a) || !PyBool_Check(b))
        return PyLong_Type.tp_as_number->nb_or(a, b);
    return PyBool_FromLong((a == Py_True) | (b == Py_True));
}

static PyObject *
bool_xor(PyObject *a, PyObject *b)
{
    if (!PyBool_Check(a) || !PyBool_Check(b))
        return PyLong_Type.tp_as_number->nb_xor(a, b);
    return PyBool_FromLong((a == Py_True) ^ (b == Py_True));
}

You can fix it by:

static PyObject *
bool_invert(PyObject *v)
{
    return bool_xor(v, Py_True);
}

(I am aware it may not be as simple as that, but I guess I am pretty close ...)
There may be more work to do as e.g.

>>> ~False & True
1

I would have expected to see True as the answer not 1, but as the ~ results in an int we get an 'int' regardless that we supplied two bool

This fix will break far less code than the deprecation, if any at all; certainly not @zhijieshi 's
Like I said before: there should be no one in the whole world relying on ~False returning -1 and ~True returning -2

[timhoffm]

@josyb thanks for your feedback. Allow me to answer on a higher level. I think we're drifting too much into details. Yes, the current implementation is flawed. (*)

I am quite sure that the followers of ~ will have encountered that; in fact @zhijieshi has, so he added the and 1 to protect himself.

Good job @zhijieshi, but I'm not that optimistic. It's easy to write if ~state:, which unexpectedly is always fulfilled. You'll find lots of such cases digging through https://github.com/search?q=language%3Apython+%2Fif+%7E%5Cw%2F&type=code. APIs must be easy to use right and hard to use wrong.

Starting from the current bool implementation, we have three options:

1. leave everything as is
2. change the behavior so that ~True is False
3. deprecate and remove ~ on bool

All three options have their downsides which have been discussed in #82012: (1) is prone to misuse and can lead to unnoticted incorrect results; (2) has conceptual arguments related to expected behavior as a int subclass against it. Also, changing behavior is risky: If somebody relied on the exact behavior - rare but not to be excluded - they will get a different (=wrong) result, possibly without noticing (errors should never pass silently). OTOH if you remove the behavior (3), any previous users will be informed. We assumed that the majority of these are incorrect uses and intended correct usage is rare.

So there are trade offs to be made. I'm unclear whether your case makes a difference in the conclusion. That's beyond my level and I leave that to the core devs.

---

(*) Theoretical background: The issues we see are caused by a variant of the circle-ellipse problem. The base class int implements ~ as bit inversion of the number in two's complement representation, or mathematically equivalent: $x \rightarrow -(x+1)$. This is reasonable (even though the actual bit representation is different due to infinite precision, but that does not matter here). The problem occurs now for the derived class bool: We logically consider bool as equivalent to an unsigned 1-bit integer. But the inherited two's complement semantics of ~ assumes signed integers, in particular $~1 = -2$. We cannot simultaneously fulfill the Liskov principle for bool as an int subclass, and maintain the unsigned 1-bit integer semantics for bool. To put it simple: bool is not suited as a subclass for int.

[josyb]

@zhijieshi

Using bitwise operations has some advantages over logical operations, e.g., in the context of MyHDL package, which is for hardware design. Not matter how many bits are in signals, we can write the same expression. We just need to mask out unnecessary bits in the end.

output_1bit = (a & ~b) & 1
output_4bits = (x & ~y) & 0xF

There should be no need to mask out the unused bits; intbv handles that by itself but unfortunately the bool doesn't.

@timhoffm
Sorry, I am a but low level guy; making your hardware ...

You'll find lots of such cases digging through https://github.com/search?q=language%3Apython+%2Fif+%7E%5Cw%2F&type=code.

I checked several examples of that list. It looks to me that they all test for the actual variable being 0. They get away by using the ~ although I agree 100% that should have used not. They will all be (rudely) awakened by the deprecation message.
(being a positive guy?) I actually most of the times avoid using not and write code like:

if s.some_condition:
   # a comment why there is nothing to do here (which incidentally also documents what is done in the other case)
   pass
else:
  do_this()

in stead of:

if not s.some_condition:
    do_this()

OTOH if you remove the behavior (3), any previous users will be informed.

Same can be said for 2. -> you could do something fundamental: remove the int base class from bool and inform any user of ~something_bool that the behavior will change in the future. The user can inspect the code and decide how or what to mitigate. I understand that this too much to ask :)

Let's try to wind down.
The final outcome is that class bool is flawed. But we can get away by restricting the use of it.
Of course this decision ripples down to MyHDL (in our case ).
If we desperately want to keep the ~, &, | and ^ operators for single bit values we would have to implement a new class bit to replace bool but that would break all MyHDL code. Reminds me of the infamous Python 2 to 3 step ...
And it will have an impact on the simulation speed, as I expect that the CPython bool will be faster than a Python class bit
I am a strong believer in not breaking code - something where the open source has no issue with. So we will have to stick with bool and the deprecation message in 3.12. I am slightly worried
I have always felt that the original MyHDL author and later maintainers made a wrong decision to use as much as possible native Python objects to represent hardware constructs - the flaw in bool underlines this. Funny that it took more than 20 years to show up, both for MyHDL as for Python itself...

[timhoffm]

You'll find lots of such cases digging through https://github.com/search?q=language%3Apython+%2Fif+%7E%5Cw%2F&type=code.

I checked several examples of that list. It looks to me that they all test for the actual variable being 0. They get away by using the ~ although I agree 100% that should have used not. They will all be (rudely) awakened by the deprecation message.

The majority of the above cases is likely valid and not affected: Note that e.g. numpy provides its own np.bool_ type, which interprets ~ as negation, so that e.g. ~np.isfinite(5) is not affected. However, there are obvious cases like this and this, which hopefully will be awakened because that code is broken.

OTOH if you remove the behavior (3), any previous users will be informed.

Same can be said for 2. -> you could do something fundamental: remove the int base class from bool and inform any user of ~something_bool that the behavior will change in the future. The user can inspect the code and decide how or what to mitigate.

It's not quite the same. Very few people read release notes and quite some people ignore warnings. So an announced change may well slip through unnoticed. While ignoring warnings is a flaw, we try to also care for the less able/experienced users. Eventually breaking code is the ultimate way to force them to take notice. - In that sense, removing functionality is safer than changing behavior.

If we desperately want to keep the ~, &, | and ^ operators for single bit values we would have to implement a new class bit to replace bool but that would break all MyHDL code.

Not necessarily. There might be other ways around. As the author of the deprecation, I'd be willing to take a look at that, if you point me to two or three representative examples that now give warnings. But that's getting off-topic here. Let's do that in myhdl/myhdl#429.

[josyb]

It's not quite the same. Very few people read release notes and quite some people ignore warnings. So an announced change may well slip through unnoticed.

I meant issue a Functionality Change Warning like the Deprecation Warning not just adding it to the release notes.

Not necessarily. There might be other ways around. As the author of the deprecation, I'd be willing to take a look at that, if you point me to two or three representative examples that now give warnings.

I read the mail before going to bed and between sleeping and waking found a possible way out: as in MyHDL single hardware connections are declared as either Signal(False) or Signal(bool(0)) we could silently replace the bool by a new bit class; the only pitfall may be the ad hoc use of a bool as a variable but there probably aren't that many users practicing this. And maybe we adopt a Deprecation Warning. It will be a bit of work, though. And we might just accept the reduced functionality of bool.
So far the only example is the one produced by @zhijieshi, but perhaps one or two will pop up when we test the MyHDL package against Python3.12 which will happen soon.

As you indicate we can close here.

Best regards,
Josy

P.S. I started checking MyHDL against 3.12, and get quite a few deprecation warnings ...

[pochmann3]

Since there is no real use-case for the current behavior

I can hardly imagine anyone relying on ~bool(0) returning -1 and ~bool(1) returning -2

In contrast, I can hardly imagine any intended use for ~some_bool, so

Like I said before: there should be no one in the whole world relying on ~False returning -1 and ~True returning -2

You've broken indexing. I've just been bitten by this change.

I sometimes want to get the first or second value of a list based on a Boolean condition. Naturally, I use mylist[cond].

And I sometimes want to index from the end. Python supports negative indexes, so naturally I use mylist[~0] for the last element and in general mylist[~i] for index i from the back. This is very useful.

Now I needed both, so naturally I used mylist[~cond]. And got a DeprecationWarning. Now I need to write mylist[~int(cond)] for no good reason? I'm not happy.

[timhoffm]

Thanks for the feedback. I‘m sorry this affects your code. mylist[cond] and mylist[~i] are unambiguous and not affected by the deprecation.

I regard the inferred mylist[~cond] as problematic. While the current implementation „bitwise inversion of the underlying int“ is what you wanted, there are numerous examples that users have interpreted ~ as boolean negation. They would write mylist[~cond] to try and get the 2nd or first element of the list (and not necessarily realize that‘s not what they get). Also they would interpret your technically correct code wrongly. In the face of these issues, I still believe that the depreciation is justified and it‘s better to be explicit and write mylist[~int(cond)] when you need it. I‘m sorry for the inconvenience that you have to adapt your code, but it’s in the interest of preventing potential misuse of the pattern and making Python a more intuitive and safe language for all.

[bjorn-martinsson]

I have two things I would like to say.

Firstly, from reading the comments here, I feel like very little consideration was taken into people that actually do make use of the ~ operator in their code. I'm here because my code broke because of this change. A long time feature of Python has been that you have always can interchangably use Booleans and integers, for example isinstance(True, int) is True. This change creates an awkward tear in this close relationship between integers and Booleans. I've identified 3 cases where my code breaks. Code involving bitmasks, code making use of ~ for "reverse indexing" of a list, and the codegolf trick of using -~x to increment x by 1.

Secondly, to anyone whose code broke because of this change. The easiest workaround that I've found is to switch out all ~ with ~+. This can safely be done with a simple search replace. The reason why this fixes the issue is that the unary plus operator converts Booleans into int, while not affecting ints (also it doesn't affect more exotic data types like np.uint8). Another benefit of using this workaround over ~int(x) is that ~int(x) can cause "silent errors" in the case where x happens to be for example a float.

[timhoffm]

Thanks for the feedback. I'm sorry that this broke your code. Let me assure you that the effect of this change has been carefully considered. There were good reasons to change and also good reasons not to change. In the end, it was a trade-off decision. Breaking some rare justified usage and violating the Liskov substitution principle vs. having an API that is prone to misuse and whose misuse is not easily detected. See #82012 (comment)

The broken cases I've seen so far were either real bugs or cases that were technically correct but could be written more clearly without bitwise inversion of bools. I'm happy to discuss your cases if you want to.

[vstinner]

See also the discussion https://discuss.python.org/t/bool-deprecation/62232