Skip to content

Pull requests: EleutherAI/lm-evaluation-harness

New pull request New
165 Open 1,660 Closed
165 Open 1,660 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Math 500
#3381 opened Nov 1, 2025 by seldereyy Loading…
1
[fix] crows_pairs dataset
#3378 opened Oct 31, 2025 by jannalulu Loading…
[feat] add graphwalks
#3377 opened Oct 31, 2025 by jannalulu Loading…
1
Fix descriptions in the Moral Stories and Histoires Morales tasks.
#3374 opened Oct 28, 2025 by upunaprosk Loading…
Fix: Prevent infinite loop when max_seq_lengths < 4096 in prepare_niah.py
#3372 opened Oct 28, 2025 by vnayakde Loading…
2
Add support for configurable chrF metric parameters in task YAML, fix…
#3363 opened Oct 23, 2025 by augustlakia Loading…
3
fix trust_remote_code=True for longbench
#3361 opened Oct 22, 2025 by jannalulu Loading…
5
Longbench group fix
#3359 opened Oct 22, 2025 by jannalulu Loading…
1
4
Fix issue 3355 assertion error
#3356 opened Oct 20, 2025 by marksverdhei Loading…
1
2
Add gsm_symbolic and gsm_symbolic_cot tasks
#3354 opened Oct 19, 2025 by MengAiDev Loading…
1
[AIME24 | AIME25] Enable Multiple Generation Repeats with Pass@k and Majority@k Metrics
#3351 opened Oct 17, 2025 by ihebchaa Loading…
1
fix(tasks):pin correct MMLUSR version
#3350 opened Oct 16, 2025 by christinaexyou Loading…
1
added azure openai support
#3349 opened Oct 16, 2025 by zinccat Loading…
1
3
Delegate BOS to the tokenizer; add_bos_token defaults to None
#3347 opened Oct 15, 2025 by baberabb Loading…
1
7
Added ULQA benchmark
#3340 opened Oct 13, 2025 by keramjan Loading…
1
Fix PIL image hashing to use actual bytes instead of object repr
#3331 opened Oct 7, 2025 by tboerstad Loading…
feat: Add support for accelerate-wrapped models in simple_evaluate()
#3313 opened Sep 26, 2025 by DhruvaKashyap Loading…
2
Add MATH500
#3311 opened Sep 26, 2025 by jannalulu Loading…
1
Support empty response for Completions and ChatCompletions API
#3309 opened Sep 22, 2025 by tboerstad Loading…
5
Adding New Task SLR-Bench : Scalable Logical Reasoning Benchmark
#3305 opened Sep 20, 2025 by Ahmad21Omar Loading…
1
Support torchrun vllm DP
#3304 opened Sep 19, 2025 by luccafong Loading…
1
Gemini evaluation support
#3300 opened Sep 15, 2025 by IsraelAbebe Loading…
Fix lambada_multilingual_stablelm
#3294 opened Sep 11, 2025 by jmichaelov Loading…
1
Adding SPaRC to lm eval harness
#3262 opened Aug 25, 2025 by lkaesberg Loading…
2
Add long-context evaluation benchmarks (LongBench v2, Babilong, InfiniteBench, Phonebook)
#3256 opened Aug 21, 2025 by Mariani-code Loading…
1
11
ProTip! What’s not been updated in a month: updated:<2025-10-03.