[Request impl] Devtool end-to-end tests

[Gasoonjia]

🚀 The feature, motivation and pitch

Devtools now have multiple components, like etdump, etrecrod, inspector, etc. Inpsector should take etdump and etrecord (optional) as input, but now we only have tests for each module separately, but do not have end-to-end tests, which may introduce some risk for our debugging pipeline. We would love for you to help us build end-to-end testing.

RFC

The overall picture of the e2e test will be like:

1. Export a simple module (e.g. an module with only add or multiple operation) and dump into buffer; using etrecord to record information; see here for example but maybe use current export flow
2. Using pybinding to load the buffer and equip with etdump
3. Propagate the etrecord and etdump to inspector api ; choose one or two inspector functions (e.g. to_dataframe) and compare the result with your expectation.

This doc might be a good resource for you to understand the end2end pipeline better.

[HonestDeng]

It's an interesting and helpful issue for me to start. Would it be alright if I take this?

[Gasoonjia]

Sure!

[HonestDeng]

Hey bro! I've seen a directory named directory/test_end2end.py.

I was thinking of creating a new file in this directory to add a unittest class. Within this class, I plan to write methods to create etdump, etrecord, and inspector. Does this approach sound reasonable?

[Gasoonjia]

@HonestDeng Thanks for your response!

directory/test_end2end.py

Which directory here are you working on? I think executorch/devtools/test/test_end2end.py would be great.

Does this approach sound reasonable?

Yeah as long as they will be composed into a single e2e test pipeline that should be fine. Target of e2e test will be testing the pipeline as a whole, rather than testing them as individual module

[HonestDeng]

Oh no, my bad! I mistakenly wrote directory/test_end2end.py. I actually meant executorch/test/end2end.

I'll implement it in executorch/devtools/test/test_end2end.py. Thanks for your response!

[HonestDeng]

Based on my reading of the documentation, here is my proposed approach:

1. Model: Prepare a simple nn.Module.
2. ETRecord: Generate the etrecord using the Python code provided in the docs.
3. ETDump: According to the docs, a bundled_program is needed to generate the etdump. The etdump itself is generated by C++ code (e.g., examples/devtools/example_runner.cpp). Therefore, after generating the bundled_program in Python, a C++ file with Python bindings (pybinding) is required. This allows the Python code to invoke the C++ function to generate the etdump.
4. Inspector: Generate the Inspector artifact using the Python code specified in the docs.

Finally, assertions, such as self.assertEqual, are needed to verify that the test executed successfully.

[Gasoonjia]

Thanks for sharing! The overall logic looks great!
For the ETDump step, you can directly use our pybinding api to generate etdump: https://github.com/pytorch/executorch/blob/main/extension/pybindings/pybindings.pyi#L151

[HonestDeng]

Hey team, I've successfully generated the etrecord and bundled_program. However, I'm having trouble figuring out how to generate the etdump.

I looked at the _load_for_executorch_from_bundled_program function in pybindings.pyi (link) and tried setting enable_etdump=True. But I'm confused – where does the etdump file get generated or saved when using this approach?

I also reviewed the C++ code in examples/devtools/example_runner.cpp (link). This example seems to imply that I need to actually execute the module to generate the etdump.

If execution is indeed required, how can I achieve this using the Python bindings? Specifically, how do I run the loaded module via Python in a way that triggers the etdump generation?

Thanks a lot!

[Gasoonjia]

you can use write_etdump_result_to_file function https://github.com/pytorch/executorch/blob/main/extension/pybindings/pybindings.cpp#L1119 to generate etdump file as well as its debug buffer file to the address you want.

[HonestDeng]

I build the project using

cmake .. -DEXECUTORCH_BUILD_DEVTOOLS=ON -DET_EVENT_TRACER_ENABLED=ON

Then try to run:

python devtools/test/test_end2end.py

But got:

[program.cpp:136] InternalConsistency verification requested but not available [pybindings.cpp:821] No etdump data found, try rebuilding with the CMake option EXECUTORCH_ENABLE_EVENT_TRACER or with buck run --config executorch.event_tracer_enabled=true

I also try to use:

cmake .. -DEXECUTORCH_BUILD_DEVTOOLS=ON -DEXECUTORCH_ENABLE_EVENT_TRACER=ON

But still failed.

why? Here is the commit e449350

[Gasoonjia]

@HonestDeng thansk for sharing
I don't think there's any issue with your comment, maybe you can try:

1. add message command in the CMakeLists.txt to print the value of this option to verify it is being set to ON.
2. clean and rebuild

Also left some comment in your commit

Please let me know if anything i can help

[HonestDeng]

I'm stuck because I'm encountering the following error:
No etdump data found, try rebuilding with the CMake option EXECUTORCH_ENABLE_EVENT_TRACER or with buck run --config executorch.event_tracer_enabled=true
I want to debug the underlying C++ code to understand what's going wrong, but I'm not sure how to do this effectively when using pybind11.
My current approach is to add print statements in the C++ code. However, I've found that I need to completely reinstall Executorch from the source (pip install -e .) every time I make a small change to the C++ code, which is very time-consuming.
Could anyone suggest a better way to debug the C++ code in this Python/C++ environment? I'm not familiar with using debuggers like GDB in this context.

[HonestDeng]

    def generate_etrecord_(self):
        aten_model: ExportedProgram = export(
            self.model,
            (torch.randn(1, 1, 32, 32), torch.randn(1, 1, 32, 32)),
        )
        edge_program_manager = to_edge(
            aten_model,
            compile_config=EdgeCompileConfig(
                _use_edge_ops=False,
                _check_ir_validity=False,
            ),
        )
        edge_program_manager_copy = copy.deepcopy(edge_program_manager)
        et_program_manager = edge_program_manager.to_executorch()

        self.et_program_manager = et_program_manager

        generate_etrecord(self.etrecord_path, edge_program_manager_copy, et_program_manager)

    def generate_etdump(self):
        # load executorch program from buffer, and set enable_etdump to True
        program = _load_for_executorch_from_buffer(self.et_program_manager.buffer, enable_etdump=True)
        # run program with example inputs to generate etdump
        program.forward((torch.randn(1, 1, 32, 32), torch.randn(1, 1, 32, 32)))
        # write etdump to file
        program.write_etdump_result_to_file(self.etdump_path)

I trid to don't use bundled program. But it still failed:
No etdump data found, try rebuilding with the CMake option EXECUTORCH_ENABLE_EVENT_TRACER or with buck run --config executorch.event_tracer_enabled=true

Is there any problem with my code?

[HonestDeng]

Currently, I installed the executorch from source using pip install -e . . Then run python test_end2end.py.
Maybe the problem is that DEXECUTORCH_BUILD_DEVTOOLS and DEXECUTORCH_ENABLE_EVENT_TRACER are not set to ON when run pip?
I'm not sure.

[Gasoonjia]

Interesting i didn;t got the issue before.
Can you commit your latest update and I can help you debugging it.

[HonestDeng]

I've committed my demo code to github. Thanks!

[metascroy]

Is this issue done @HonestDeng ?

[Gasoonjia]

@metascroy nope
@HonestDeng thanks bro let me invest your code