Skip to content

Commit 1244b00

Browse files
committed
Update comment
1 parent a06e82b commit 1244b00

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

tests/test-tokenizer-1-bpe.cpp

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ int main(int argc, char **argv) {
9191
}
9292
}
9393
}
94-
// TODO: why doesn't this work for the full range of Unicodes?
94+
// Restrict to unicode planes
9595
// for (uint32_t cp = 0x10000; cp < 0x0010ffff; ++cp) {
9696
for (uint32_t cp = 0x10000; cp < 0x00040000; ++cp) {
9797
std::string str = codepoint_to_utf8(cp);

0 commit comments

Comments
 (0)