bpo-29591: Upgrade Modules/expat to libexpat 2.2 #2164

vstinner · 2017-06-13T14:45:00Z

[Security] bpo-29591: Update expat copy from 2.1.1 to 2.2.0 to get fixes of CVE-2016-0718 and CVE-2016-4472. See https://sourceforge.net/p/expat/bugs/537/ for more information.

vstinner · 2017-06-13T14:48:47Z

Modules/expat/expat.h

@@ -342,7 +342,7 @@ XML_SetEntityDeclHandler(XML_Parser parser,
                         XML_EntityDeclHandler handler);

 /* OBSOLETE -- OBSOLETE -- OBSOLETE
-   This handler has been superseded by the EntityDeclHandler above.
+   This handler has been superceded by the EntityDeclHandler above.


wait, superseded is the right spelling no?

it is... but lets not correct spelling in third party code.

oh, i see, this was a python tree only change? weird. keep it as superseded.

it is... but lets not correct spelling in third party code.

Right, I prefer to reduce the downstream changes: so I reverted this change as well.

vstinner · 2017-06-13T14:49:21Z

Modules/expat/amigaconfig.h

@@ -7,6 +7,9 @@
 /* Define to 1 if you have the `bcopy' function. */
 #define HAVE_BCOPY 1

+/* Define to 1 if you have the <check.h> header file. */
+#undef HAVE_CHECK_H


I don't understand this change, I just tried to reduce the changes.

we don't even need to carry this file around given the #elif defined(__amigaos__) it is included under.

I'm not sure that I understood. Do you want to remove amigaconfig.h?

In the meanwhile, I reverted this change.

vstinner · 2017-06-13T14:49:58Z

Modules/expat/expat_external.h

@@ -7,6 +7,10 @@

 /* External API definitions */

+/* Namespace external symbols to allow multiple libexpat version to
+   co-exist. */
+#include "pyexpatns.h"


I would be nice to upstream pyexpatns.h namespace, it seems useful to build a clean DLL on Windows.

gpshead

I believe we can drop amigaconfig.h, watcomconfig.h, and macconfig.h (used for MacOS classic on PPC32).

matrixise · 2017-06-13T16:32:21Z

Duplicate of #2021 ?

vstinner · 2017-06-13T21:44:36Z

@matrixise: "Duplicate of #2021 ?"

I chose a different approach. I wanted to recreate Modules/expat/ directory "from scratch", so I wrote a shell script that I attached http://bugs.python.org/issue29591

I tried to restore Python changes reverted by my approach, to more easily identify downstream (Python) changes.

I compare my PR with your (rebased) PR:

you didn't update COPYING
you removed: #include "pyexpatns.h"
you removed: #define XML_HAS_SET_HASH_SALT
my PR now also removes amigaconfig.h, macconfig.h and watcomconfig.h of Modules/expat/

gpshead · 2017-06-13T22:50:59Z

FWIW I agree with recreating the directory from scratch and applying the few actually relevant local changes on top as you have done.

gpshead · 2017-06-13T22:54:01Z

We can remove the XML_HAS_SET_HASH_SALT define and remove that from the ifdef in Modules/pyexpat.c as that was a custom Python tree only addition I did back when I backported those changes to the old version of expat we had at the time.

Remove the configuration (Modules/expat/*config.h) of unsupported platforms: * Amiga * MacOS Classic on PPC32 * Open Watcom

The XML_HAS_SET_HASH_SALT define of Modules/expat/expat.h became useless since our local expat copy was upgrade to expat 2.1 (it's now expat 2.2.0).

vstinner · 2017-06-14T10:05:27Z

@gpshead: "We can remove the XML_HAS_SET_HASH_SALT define and remove that from the ifdef in Modules/pyexpat.c (...)"

Right, done.

" I believe we can drop amigaconfig.h, watcomconfig.h, and macconfig.h (used for MacOS classic on PPC32)." => done

vstinner · 2017-06-14T12:28:09Z

Does it look good to you now @gpshead?

tiran

Your patch on top of expat upstream looks good to me. I don't have enough resources to properly verify all changes in expat, though.

vstinner · 2017-06-14T22:54:51Z

Thanks for the reviews @tiran and @gpshead!

* bpo-29591: Upgrade Modules/expat to libexpat 2.2 * bpo-29591: Restore Python changes on expat * bpo-29591: Remove expat config of unsupported platforms Remove the configuration (Modules/expat/*config.h) of unsupported platforms: * Amiga * MacOS Classic on PPC32 * Open Watcom * bpo-29591: Remove useless XML_HAS_SET_HASH_SALT The XML_HAS_SET_HASH_SALT define of Modules/expat/expat.h became useless since our local expat copy was upgrade to expat 2.1 (it's now expat 2.2.0). (cherry picked from commit 23ec4b5)

…2164) (#2203) * bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) * bpo-29591: Upgrade Modules/expat to libexpat 2.2 * bpo-29591: Restore Python changes on expat * bpo-29591: Remove expat config of unsupported platforms Remove the configuration (Modules/expat/*config.h) of unsupported platforms: * Amiga * MacOS Classic on PPC32 * Open Watcom * bpo-29591: Remove useless XML_HAS_SET_HASH_SALT The XML_HAS_SET_HASH_SALT define of Modules/expat/expat.h became useless since our local expat copy was upgrade to expat 2.1 (it's now expat 2.2.0). (cherry picked from commit 23ec4b5) * bpo-30694: Upgrade Modules/expat/ to libexpat 2.2.1 (#2300) New file: Modules/expat/siphash.h. (cherry picked from commit 5ff7132) * bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348) bpo-30726, bpo-29591: libexpat 2.2.1 of Modules/expat/ now uses a winconfig.h configuration file which already defines: * XML_NS * XML_DTD * BYTEORDER=1234 * XML_CONTEXT_BYTES=1024 * HAVE_MEMMOVE Remove these defines from PCbuild/_elementtree.vcxproj to prevent compiler warnings. Co-Authored-By: Jeremy Kloth <[email protected]> (cherry picked from commit c8fb58b) * bpo-30726: Fix elementtree warnings on Windows due to expat upgrade (#2319) * bpo-30726: Fix elementtree warnings on Windows Caused by usage of `getenv` which should be safe. And a few integer truncations which should also be ok. * bpo-30726: Don't ignore libexpat warnings which haypo intends to fix upstream (cherry picked from commit 87c6555)

…2164) (#2204) * bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) * bpo-29591: Upgrade Modules/expat to libexpat 2.2 * bpo-29591: Restore Python changes on expat * bpo-29591: Remove expat config of unsupported platforms Remove the configuration (Modules/expat/*config.h) of unsupported platforms: * Amiga * MacOS Classic on PPC32 * Open Watcom * bpo-29591: Remove useless XML_HAS_SET_HASH_SALT The XML_HAS_SET_HASH_SALT define of Modules/expat/expat.h became useless since our local expat copy was upgrade to expat 2.1 (it's now expat 2.2.0). (cherry picked from commit 23ec4b5) * bpo-30694: Upgrade Modules/expat/ to libexpat 2.2.1 (#2300) New file: Modules/expat/siphash.h. (cherry picked from commit 5ff7132) * bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348) bpo-30726, bpo-29591: libexpat 2.2.1 of Modules/expat/ now uses a winconfig.h configuration file which already defines: * XML_NS * XML_DTD * BYTEORDER=1234 * XML_CONTEXT_BYTES=1024 * HAVE_MEMMOVE Remove these defines from PCbuild/_elementtree.vcxproj to prevent compiler warnings. Co-Authored-By: Jeremy Kloth <[email protected]> (cherry picked from commit c8fb58b) * bpo-30726: Fix elementtree warnings on Windows due to expat upgrade (#2319) * bpo-30726: Fix elementtree warnings on Windows Caused by usage of `getenv` which should be safe. And a few integer truncations which should also be ok. * bpo-30726: Don't ignore libexpat warnings which haypo intends to fix upstream (cherry picked from commit 87c6555)

vstinner added the type-security A security issue label Jun 13, 2017

vstinner requested review from gpshead, benjaminp and tiran June 13, 2017 14:45

the-knights-who-say-ni added the CLA signed label Jun 13, 2017

vstinner commented Jun 13, 2017

View reviewed changes

gpshead previously requested changes Jun 13, 2017

View reviewed changes

vstinner added 4 commits June 14, 2017 11:58

bpo-29591: Upgrade Modules/expat to libexpat 2.2

579e473

bpo-29591: Restore Python changes on expat

01d6002

bpo-29591: Remove expat config of unsupported platforms

c6d91b9

Remove the configuration (Modules/expat/*config.h) of unsupported platforms: * Amiga * MacOS Classic on PPC32 * Open Watcom

bpo-29591: Remove useless XML_HAS_SET_HASH_SALT

d89fffb

The XML_HAS_SET_HASH_SALT define of Modules/expat/expat.h became useless since our local expat copy was upgrade to expat 2.1 (it's now expat 2.2.0).

matrixise mentioned this pull request Jun 14, 2017

bpo-30610: [Security] Python's libexpat vulnerable to CVE-2016-0718 #2021

Closed

tiran approved these changes Jun 14, 2017

View reviewed changes

vstinner merged commit 23ec4b5 into python:master Jun 14, 2017

vstinner deleted the expat_22 branch June 14, 2017 22:54

smurav mentioned this pull request Nov 26, 2024

gh-127285: Fix ARM platform configuration in pcbuild.sln #127286

Merged

Uh oh!

bpo-29591: Upgrade Modules/expat to libexpat 2.2 #2164

bpo-29591: Upgrade Modules/expat to libexpat 2.2 #2164

Uh oh!

Conversation

vstinner commented Jun 13, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gpshead left a comment

Choose a reason for hiding this comment

Uh oh!

matrixise commented Jun 13, 2017

Uh oh!

vstinner commented Jun 13, 2017

Uh oh!

gpshead commented Jun 13, 2017

Uh oh!

gpshead commented Jun 13, 2017

Uh oh!

vstinner commented Jun 14, 2017

Uh oh!

vstinner commented Jun 14, 2017

Uh oh!

tiran left a comment

Choose a reason for hiding this comment

Uh oh!

vstinner commented Jun 14, 2017

Uh oh!

Uh oh!