Skip to content

feat: MarkdownHeaderSplitter #9660

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 85 commits into
base: main
Choose a base branch
from
Open

feat: MarkdownHeaderSplitter #9660

wants to merge 85 commits into from
+771 −0

Conversation

@OGuggenbuehl
Copy link

@OGuggenbuehl OGuggenbuehl commented Jul 29, 2025 •
edited
Loading

Proposed Changes:

Implement MarkdownHeaderSplitter to split Documents written in .md based on their headers

How did you test it?

unit tests

Checklist

  • I have read the contributors guidelines and the code of conduct
  • I have updated the related issue with new insights and changes
  • I added unit tests and updated the docstrings
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I documented my code
  • I ran pre-commit hooks and fixed any issue

@CLAassistant
Copy link

CLAassistant commented Jul 29, 2025 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions bot added topic:tests type:documentation Improvements on the docs labels Jul 29, 2025
@OGuggenbuehl OGuggenbuehl changed the title Feature/md header splitter feat:MarkdownHeaderSplitter Jul 29, 2025
@sjrl sjrl self-assigned this Aug 19, 2025
sjrl
sjrl reviewed Aug 19, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 19, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 19, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 19, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 19, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 19, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 19, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 19, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 19, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 19, 2025
View reviewed changes
@sjrl
Copy link
Contributor

sjrl commented Aug 19, 2025

@OGuggenbuehl definitely looks like an interesting approach! I've left an initial set of comments, but to further review I'd appreciate if you could add a set of tests like the ones we have for the DocumentSplitter https://github.com/deepset-ai/haystack/blob/main/test/components/preprocessors/test_document_splitter.py

This will help me be able to review the actual algorithm for splitting since it's easier to understand with examples.

@sjrl sjrl changed the title feat:MarkdownHeaderSplitter feat: MarkdownHeaderSplitter Aug 27, 2025
@OGuggenbuehl OGuggenbuehl force-pushed the feature/md-header-splitter branch from 61a8396 to bcbbf9a Compare September 16, 2025 13:57
@sjrl
Copy link
Contributor

sjrl commented Sep 18, 2025 •
edited
Loading

Thanks for your continued work on this @OGuggenbuehl!

Some general comments. Could you:

  • Add a release note for this PR following the instructions here
  • Could you make sure to include our license header to the beginning of each file you've added. You can find an example of the license header here
  • Please make sure to sign the CLA agreement (docs about it here) from this comment
  • If you haven't already please also set up pre-commit hooks using pre-commit install. You can find more info about that in this section of our contribution guidelines.
  • Also in the future feel free to open branches directly in Haystack instead of using a fork. This makes it slightly easier to pull down your code to review locally.

sjrl
sjrl reviewed Sep 18, 2025
View reviewed changes
sjrl
sjrl reviewed Sep 18, 2025
View reviewed changes
sjrl
sjrl reviewed Sep 18, 2025
View reviewed changes
sjrl
sjrl reviewed Sep 18, 2025
View reviewed changes
sjrl
sjrl reviewed Sep 18, 2025
View reviewed changes
@coveralls
Copy link
Collaborator

coveralls commented Sep 19, 2025 •
edited
Loading

Pull Request Test Coverage Report for Build 19816136481

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 3 unchanged lines in 1 file lost coverage.
  • Overall coverage remained the same at 92.189%
Files with Coverage Reduction New Missed Lines %
core/pipeline/async_pipeline.py 3 65.88%
Totals Coverage Status
Change from base Build 19775495543: 0.0%
Covered Lines: 14174
Relevant Lines: 15375
💛 - Coveralls

@OGuggenbuehl OGuggenbuehl marked this pull request as ready for review September 19, 2025 16:05
@OGuggenbuehl OGuggenbuehl requested review from a team as code owners September 19, 2025 16:05
@OGuggenbuehl OGuggenbuehl removed the request for review from a team September 19, 2025 16:05
OGuggenbuehl and others added 29 commits December 1, 2025 09:28
@OGuggenbuehl
return doc unchunked if no headers have content
32d8c68
@OGuggenbuehl
add doc-id to logging statement for unsplit documents
bcf56ca
@OGuggenbuehl
remove unneeded logs
c5415ec
@OGuggenbuehl
minor cleanup
dff06bc
@OGuggenbuehl
simplify page-number tracking method to not return count, just the up…
a54d25a 
…dated page number
@OGuggenbuehl
add dev comment to mypy check for doc.content is None
a34c7a6
@OGuggenbuehl @sjrl
Update haystack/components/preprocessors/markdown_header_splitter.py
7bc798e 
Co-authored-by: Sebastian Husch Lee <[email protected]>
@OGuggenbuehl
remove split meta flattening
a7eef6b
@OGuggenbuehl
keep empty meta return consistent
5b5fc93
@OGuggenbuehl
remove unneeded content is none check
8ef5af0
@OGuggenbuehl
update tests to reflect empty meta dict for unsplit docs
f1e3739
@OGuggenbuehl
clean up total_page counts
df7e775
@OGuggenbuehl
remove unneeded meta check
3c1c376
@OGuggenbuehl @sjrl
Update test/components/preprocessors/test_markdown_header_splitter.py
86feef6 
Co-authored-by: Sebastian Husch Lee <[email protected]>
@OGuggenbuehl
implement keep_headers parameter
c22b57d
@OGuggenbuehl
remove meta-dict flattening
7c03a04
@OGuggenbuehl
add minor sanity checks
9a8ca76
@OGuggenbuehl @sjrl
Update test/components/preprocessors/test_markdown_header_splitter.py
2f1e203 
Co-authored-by: Sebastian Husch Lee <[email protected]>
@OGuggenbuehl
add warmup
b22feb5
@OGuggenbuehl @sjrl
Update haystack/components/preprocessors/markdown_header_splitter.py
8501831 
Co-authored-by: Sebastian Husch Lee <[email protected]>
@OGuggenbuehl
fix splitting when keeping headers
23da68e
@OGuggenbuehl
test cleanup to cover keep_headers=True
ccc1057
@OGuggenbuehl
add tests for keep_headers=False splitting
c4a5c17
@OGuggenbuehl
remove strip()
f3d7799
@OGuggenbuehl
simplify doc handling
f842fdb
@OGuggenbuehl
fix split id assignment
c7fc2e4
@OGuggenbuehl
test cleanup
64ff6fb
@OGuggenbuehl
test splits more explicitly
eb3e568
@OGuggenbuehl
cleanup tests
ad155cc 
minor commenting
@OGuggenbuehl OGuggenbuehl force-pushed the feature/md-header-splitter branch from c7264e6 to ad155cc Compare December 1, 2025 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dfokina dfokina Awaiting requested review from dfokina dfokina was automatically assigned from deepset-ai/documentation

@sjrl sjrl Awaiting requested review from sjrl

At least 1 approving review is required to merge this pull request.

Assignees

@sjrl sjrl

Labels

topic:build/distribution topic:CI topic:core topic:DX Developer Experience topic:tests type:documentation Improvements on the docs

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@OGuggenbuehl @CLAassistant @sjrl @coveralls