Skip to content

closes bpo-39926: Update Unicode to 13.0.0. #18910

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 11, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Doc/library/stdtypes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -352,7 +352,7 @@ Notes:
The numeric literals accepted include the digits ``0`` to ``9`` or any
Unicode equivalent (code points with the ``Nd`` property).

See http://www.unicode.org/Public/12.1.0/ucd/extracted/DerivedNumericType.txt
See http://www.unicode.org/Public/13.0.0/ucd/extracted/DerivedNumericType.txt
for a complete list of code points with the ``Nd`` property.


Expand Down
8 changes: 4 additions & 4 deletions Doc/library/unicodedata.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@

This module provides access to the Unicode Character Database (UCD) which
defines character properties for all Unicode characters. The data contained in
this database is compiled from the `UCD version 12.1.0
<http://www.unicode.org/Public/12.1.0/ucd>`_.
this database is compiled from the `UCD version 13.0.0
<http://www.unicode.org/Public/13.0.0/ucd>`_.

The module uses the same names and symbols as defined by Unicode
Standard Annex #44, `"Unicode Character Database"
Expand Down Expand Up @@ -175,6 +175,6 @@ Examples:

.. rubric:: Footnotes

.. [#] http://www.unicode.org/Public/12.1.0/ucd/NameAliases.txt
.. [#] http://www.unicode.org/Public/13.0.0/ucd/NameAliases.txt

.. [#] http://www.unicode.org/Public/12.1.0/ucd/NamedSequences.txt
.. [#] http://www.unicode.org/Public/13.0.0/ucd/NamedSequences.txt
2 changes: 1 addition & 1 deletion Doc/reference/lexical_analysis.rst
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,7 @@ The Unicode category codes mentioned above stand for:
* *Nd* - decimal numbers
* *Pc* - connector punctuations
* *Other_ID_Start* - explicit list of characters in `PropList.txt
<http://www.unicode.org/Public/12.1.0/ucd/PropList.txt>`_ to support backwards
<http://www.unicode.org/Public/13.0.0/ucd/PropList.txt>`_ to support backwards
compatibility
* *Other_ID_Continue* - likewise

Expand Down
5 changes: 5 additions & 0 deletions Doc/whatsnew/3.9.rst
Original file line number Diff line number Diff line change
Expand Up @@ -372,6 +372,11 @@ types with context-specific metadata and new ``include_extras`` parameter to
:func:`typing.get_type_hints` to access the metadata at runtime. (Contributed
by Till Varoquaux and Konstantin Kashin.)

unicodedata
-----------

The Unicode database has been updated to version 13.0.0. (:issue:`39926`).

venv
----

Expand Down
1 change: 1 addition & 0 deletions Lib/test/test_ucn.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ def test_cjk_unified_ideographs(self):
self.checkletter("CJK UNIFIED IDEOGRAPH-2B734", "\U0002B734")
self.checkletter("CJK UNIFIED IDEOGRAPH-2B740", "\U0002B740")
self.checkletter("CJK UNIFIED IDEOGRAPH-2B81D", "\U0002B81D")
self.checkletter("CJK UNIFIED IDEOGRAPH-3134A", "\U0003134A")

def test_bmp_characters(self):
for code in range(0x10000):
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Update Unicode database to Unicode version 13.0.0.
9 changes: 5 additions & 4 deletions Modules/unicodedata.c
Original file line number Diff line number Diff line change
Expand Up @@ -1031,13 +1031,14 @@ static int
is_unified_ideograph(Py_UCS4 code)
{
return
(0x3400 <= code && code <= 0x4DB5) || /* CJK Ideograph Extension A */
(0x4E00 <= code && code <= 0x9FEF) || /* CJK Ideograph */
(0x20000 <= code && code <= 0x2A6D6) || /* CJK Ideograph Extension B */
(0x3400 <= code && code <= 0x4DBF) || /* CJK Ideograph Extension A */
(0x4E00 <= code && code <= 0x9FFC) || /* CJK Ideograph */
(0x20000 <= code && code <= 0x2A6DD) || /* CJK Ideograph Extension B */
(0x2A700 <= code && code <= 0x2B734) || /* CJK Ideograph Extension C */
(0x2B740 <= code && code <= 0x2B81D) || /* CJK Ideograph Extension D */
(0x2B820 <= code && code <= 0x2CEA1) || /* CJK Ideograph Extension E */
(0x2CEB0 <= code && code <= 0x2EBEF); /* CJK Ideograph Extension F */
(0x2CEB0 <= code && code <= 0x2EBE0) || /* CJK Ideograph Extension F */
(0x30000 <= code && code <= 0x3134A); /* CJK Ideograph Extension G */
}

/* macros used to determine if the given code point is in the PUA range that
Expand Down
4,595 changes: 2,348 additions & 2,247 deletions Modules/unicodedata_db.h

Large diffs are not rendered by default.

52,784 changes: 26,826 additions & 25,958 deletions Modules/unicodename_db.h

Large diffs are not rendered by default.

1,093 changes: 575 additions & 518 deletions Objects/unicodetype_db.h

Large diffs are not rendered by default.

9 changes: 5 additions & 4 deletions Tools/unicode/makeunicodedata.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
# * Doc/library/stdtypes.rst, and
# * Doc/library/unicodedata.rst
# * Doc/reference/lexical_analysis.rst (two occurrences)
UNIDATA_VERSION = "12.1.0"
UNIDATA_VERSION = "13.0.0"
UNICODE_DATA = "UnicodeData%s.txt"
COMPOSITION_EXCLUSIONS = "CompositionExclusions%s.txt"
EASTASIAN_WIDTH = "EastAsianWidth%s.txt"
Expand Down Expand Up @@ -100,13 +100,14 @@

# these ranges need to match unicodedata.c:is_unified_ideograph
cjk_ranges = [
('3400', '4DB5'),
('4E00', '9FEF'),
('20000', '2A6D6'),
('3400', '4DBF'),
('4E00', '9FFC'),
('20000', '2A6DD'),
('2A700', '2B734'),
('2B740', '2B81D'),
('2B820', '2CEA1'),
('2CEB0', '2EBE0'),
('30000', '3134A'),
]


Expand Down