A short term fix for editable mode install failed to import root level module such as exir (#9818)

Summary:

Fixes #9558.

The `src/executorch/exir` file exists primarily due to the limitations
of `pip install` in editable mode. Specifically, `pip install -e .` does
not recognize `<executorch root>/exir` (or any root level directory with
a `__init__.py`) as a valid package module because of the presence of
`<executorch root>/exir/__init__.py`. See the following GitHub issue for
details: [Issue
#9558](#9558).

To work around this limitation, a symlink is used. With this symlink and
this package entry in `pyproject.toml`:

```toml
[tool.setuptools.package-dir]
# ...
"executorch" = "src/executorch"
```
We are telling `pip install -e .` to treat `src/executorch` as the root
of the `executorch` package and hence mapping `executorch.exir` to
`src/executorch/exir`. This effectively gets `exir` out from the root
level package.

This allows us to perform `pip install -e .` successfully and enables
the execution of the following command:

```bash
python -c "from executorch.exir import CaptureConfig"
```

Test Plan:

```bash
./install_executorch.sh --pybind --editable
python -c "from executorch.exir import CaptureConfig"
```

Reviewers:

Subscribers:

Tasks:

Tags:

Author: larryliu0820

pyproject.toml

@@ -88,23 +88,15 @@ flatc = "executorch.data.bin:flatc"
 # package, and package_data to restrict the set up non-python files we
 # include. See also setuptools/discovery.py for custom finders.
 [tool.setuptools.package-dir]
-"executorch.backends" = "backends"
-"executorch.codegen" = "codegen"
-"executorch.data.bin" = "data/bin"
-# TODO(mnachin T180504136): Do not put examples/models
-# into core pip packages. Refactor out the necessary utils
-# or core models files into a separate package.
-"executorch.examples.apple.coreml.llama" = "examples/apple/coreml/llama"
-"executorch.examples.llm_pte_finetuning" = "examples/llm_pte_finetuning"
-"executorch.examples.models" = "examples/models"
-"executorch.exir" = "exir"
-"executorch.extension" = "extension"
-"executorch.kernels.quantized" = "kernels/quantized"
-"executorch.schema" = "schema"
-"executorch.devtools" = "devtools"
-"executorch.devtools.bundled_program" = "devtools/bundled_program"
-"executorch.runtime" = "runtime"
-"executorch.util" = "util"
+# Tell setuptools to follow the symlink: src/executorch/* -> * for all first level
+# modules such as src/executorch/exir -> exir. This helps us to semi-compliant with
+# the "src layout" convention for python packages, which is also discussed in
+# https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/.
+# In the long term we should move all the modules under the src/executorch/ folder.
+#
+# Doing this also allows us to import from executorch.exir directly in
+# editable mode.
+"executorch" = "src/executorch"
 
 [tool.setuptools.package-data]
 # TODO(dbort): Prune /test[s]/ dirs, /third-party/ dirs, yaml files that we

setup.py

@@ -575,8 +575,7 @@ def run(self):
             # In editable mode, the package directory is the original source directory
             dst_root = self.get_package_dir(".")
         else:
-            dst_root = os.path.join(self.build_lib, self.get_package_dir("executorch"))
-
+            dst_root = os.path.join(self.build_lib, "executorch")
         # Create the version file.
         Version.write_to_python_file(os.path.join(dst_root, "version.py"))
 

src/README.md

@@ -0,0 +1,26 @@
+# Why Do We Have These Symlinks
+
+The `src/executorch/*` files exist primarily due to the limitations of `pip install` in editable mode. Specifically, `pip install -e .` does not recognize `<executorch root>/exir` (or any root level directory with a `__init__.py`) as a valid package module because of the presence of `<executorch root>/exir/__init__.py`. See the following GitHub issue for details: [Issue #9558](https://github.com/pytorch/executorch/issues/9558).
+
+## The Symlink Solution
+
+To work around this limitation, a symlink is used. With this symlink and this package entry in `pyproject.toml`:
+
+```toml
+[tool.setuptools.package-dir]
+# ...
+"executorch" = "src/executorch"
+```
+We are telling `pip install -e .` to treat `src/executorch` as the root of the `executorch` package and hence mapping `executorch.*.*` to `src/executorch/*/*`. This effectively gets modules like `exir` out from the root level package.
+
+This allows us to perform `pip install -e .` successfully and enables the execution of the following command:
+
+```bash
+python -c "from executorch.exir import CaptureConfig"
+```
+
+## Long Term Solution
+
+We should start to move directories from <executorch root>/ to <executorch root>/src/ and remove the symlinks. Issue [#8699](https://github.com/pytorch/executorch/issues/8699) to track this effort. This will require a lot of work internally.
+
+TODO(mnachin T180504136): Do not put examples/models into core pip packages. Refactor out the necessary utils or core models files into a separate package.

src/executorch/backends

@@ -0,0 +1 @@
+../../backends

src/executorch/codegen

@@ -0,0 +1 @@
+../../codegen

src/executorch/data

@@ -0,0 +1 @@
+../../data

src/executorch/devtools

@@ -0,0 +1 @@
+../../devtools

src/executorch/examples/apple

@@ -0,0 +1 @@
+../../../examples/apple

src/executorch/examples/llm_pte_finetuning

@@ -0,0 +1 @@
+../../../examples/llm_pte_finetuning

src/executorch/examples/models

@@ -0,0 +1 @@
+../../../examples/models