Add script to fetch benchmark results for execuTorch

[yangw-dev]

Summary

Provide methods and script to fetch all execuTorch benchamrk data from HUD API into two dataset,private and public, the script will:

• fetch all data from HUD API from input time range in UTC
• clean out records and tables with only FAILURE_REPORT due to job-level failures
• get all private table metrics, generate table_name and find intersected public table metrics
• generate private and public table groups
• output data

OutputType:

• run with excel-sheet export
• run with csv export
• run with dataframe format print
• run with json format print

See more guidance in README.md

the data is similar to the excel sheet generated manually in #10982
The result should be the same as the hud per model datatable:

helper methods: common.py

provide common.py helper method to convert back csv and excel sheets back to {"groupInfo":{}, "df":df.DataFrame} format.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11734

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures

As of commit 99df0fe with merge base 9bb0735 ():

NEW FAILURES - The following jobs have failed:

• Build Linux Wheels / pytorch/executorch / build-manywheel-py3_10-cpu (gh)
  ModuleNotFoundError: No module named 'torchvision'
• Build Linux Wheels / pytorch/executorch / upload / upload-manywheel-py3_10-cpu (gh)
  Unable to download artifact(s): Artifact not found for name: pytorch_executorch__3.10_cpu_x86_64
• Build macOS Wheels / pytorch/executorch / build-wheel-py3_10-cpu (gh)
  ModuleNotFoundError: No module named 'torchvision'
• Build macOS Wheels / pytorch/executorch / upload / upload-wheel-py3_10-cpu (gh)
  Unable to download artifact(s): Artifact not found for name: pytorch_executorch__3.10_cpu_
• pull / test-llava-runner-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 7db129c6c587e8f72ee1db745cffc55aca1528bcf25527963711c5a4d678aa36 /exec failed with exit code 139

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[yangw-dev]

FYI, this method can be more general, but since only execuTorch is using it, i just make it execuTorch specific @huydhn

[yangw-dev]

the excel sheet has limit of sheet name len < 31, which can be easy to break in the future. @huydhn @guangy10 , I think instead of generate one file per category, maybe we can generate list of excel files stored in folders [private, public]

But right now with the hard-coded abbreviation, this works fine. THe excel sheet option is there in case people want to use it.

[huydhn]

Stamped to unblock! Let's start using the script and improve it along the way

[guangy10]

We configured a fixed list of matching names to list limited tables

I think we should make it more flexible as there are always new models, recipes, devices added. For example, we recently add more models (see on dash) from huggingface/optimum-executorch to the benchmark infra, and the list will keep expanding.

Similarly with new "devices" or "backends" available, we want to be able to query the results via the script as well.

the excel sheet has limit of sheet name len < 31, which can be easy to break in the future. @huydhn @guangy10 , I think instead of generate one file per category, maybe we can generate list of excel files stored in folders [private, public]

Yeah noticed the limits when I manually created the excel sheet. Ideally I'd like to get rid of the excel sheet by wiring the outputs from db to the analysis script directly. Given what is currently supported in this PR, what does the workflow look like if I want to rerun the analysis? That is, how is this script interfaced to the analysis script?

[yangw-dev]

@pytorchbot label "release notes: none"

[guangy10]

1. Is this field always refer to private device?
2. What is the format to specify multiple devices? comma-separated or something else, should it be clarified here?
3. All supported devices can be found here (Note it's mixing of private and public), should --help list all supported devices so users can copy/paste easily.
4. We should be consistent on names of these devices in all places. https://github.com/pytorch/executorch/blob/main/.ci/scripts/gather_benchmark_configs.py#L20-L29

[yangw-dev]

I think this can be general, I can associate with the list to provide list of choices when user try to add to it, if underscore needs to be consistent, i can change some thing, but

[yangw-dev]

I changed everything to private devices now since we first filter wiht private devices, i only add two private_device_pool options there

[guangy10]

Thank you for iterating on this! There are few critical issues to be fixed in order to merge

[yangw-dev]

@guangy10
I updated logics
Fetch and process benchmark data from HUD API for ExecutorchBenchmark.

This class provides methods to:
1. Fetch all benchmark data for a specified time range
2. Get all private device info within the time range
3. Filter the private device data if filter is provided, for backends and models we expect human-readable models, notice, everything is first based on private devices.
4. Then use the filtered private device data to find matched the public device data using [model, backend, device, arch]
3. Export results in various formats (JSON, DataFrame, Excel, CSV)