[1/3] Recognize `.py.in` and `.pyi.in` files as Python in VS Code #95200

XuehaiPan · 2023-02-21T09:51:47Z

Changes:

=> this PR: [1/3] Recognize .py.in and .pyi.in files as Python in VS Code #95200

Recognize .py.in and .pyi.in files as Python in VS Code for a better development experience.
Fix deep setting merge in tools/vscode_settings.py.

[2/3] Update .pyi Python stub files: Prettify rnn.py by using type annotated NamedTuple #95267

Use Namedtuple rather than namedtuple + __annotations__ for torch.nn.utils.rnn.PackedSequence_:

namedtuple + __annotations__:

PackedSequence_ = namedtuple('PackedSequence_',
                             ['data', 'batch_sizes', 'sorted_indices', 'unsorted_indices'])

# type annotation for PackedSequence_ to make it compatible with TorchScript
PackedSequence_.__annotations__ = {'data': torch.Tensor, 'batch_sizes': torch.Tensor,
                                   'sorted_indices': Optional[torch.Tensor],
                                   'unsorted_indices': Optional[torch.Tensor]}

Namedtuple: Python 3.6+

class PackedSequence_(NamedTuple):
    data: torch.Tensor
    batch_sizes: torch.Tensor
    sorted_indices: Optional[torch.Tensor]
    unsorted_indices: Optional[torch.Tensor]

[3/3] Update .pyi Python stub files and enable 'UFMT' linter #95268

Sort import statements and remove unnecessary imports in .pyi, .pyi.in files.
Format .pyi, .pyi.in files and remove unnecessary ellipsis ... in type stubs.

pytorch-bot · 2023-02-21T09:51:50Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/95200

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 391b2f5:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

XuehaiPan · 2023-02-21T13:28:50Z

torch/_C/_distributed_rpc.pyi

We should not use mutable variables as function defaults.

Ref: dangerous-default-value / W0102

Correct code:

def func( ..., device_maps: Optional[Dict[str, Dict[torch.device, torch.device]]] = None, devices: Optional[List[torch.device]] = None ): if device_maps is None: device_maps = {} if devices is None: devices = [] ...

albanD · 2023-02-21T14:57:50Z

Thanks for the PR!
Could you split this up into formatting-only PRs and the rest?
In particular, the PackedSequence_ change would need to be looked at carefully.

Also I'm not sure what the vscode_settings.py file does, how to test these changes?

XuehaiPan · 2023-02-22T05:21:56Z

Could you split this up into formatting-only PRs and the rest?
In particular, the PackedSequence_ change would need to be looked at carefully.

I split this into 3 PRs. The PackedSequence_ change is moved to a standalone PR.

Also I'm not sure what the vscode_settings.py file does, how to test these changes?

You could run:

./tools/vscode_settings.py

or

python3 ./tools/vscode_settings.py

It will update the workspace VS Code settings .vscode/settings.json from .vscode/settings_recommended.json. The script will create a setting file if not exist.

For example:

current file (.vscode/settings.json):

{
    "[c]": {
        "editor.insertSpaces": true,
        "editor.tabSize": 2
    },
    "[cpp]": {
        "editor.insertSpaces": true,
        "editor.tabSize": 2,
        "editor.defaultFormatter": "ms-vscode.cpptools"
    },
    "[python]": {
        "editor.defaultFormatter": "ms-python.black-formatter"
    },
    "files.associations": {
        "aten/src/ATen/ATenConfig.cmake.in": "cmake"
    },
    "java.autobuild.enabled": false,
    "java.compile.nullAnalysis.mode": "disabled",
}

after running ./tools/vscode_settings.py:

{
    "[c]": {
        "editor.insertSpaces": true,
        "editor.tabSize": 2,
    },
    "[cpp]": {
        "editor.insertSpaces": true,
        "editor.tabSize": 2,
        "editor.defaultFormatter": "ms-vscode.cpptools",
    },
    "[python]": {
        "editor.defaultFormatter": "ms-python.black-formatter",
+       "editor.tabSize": 4,
    },
    "files.associations": {
        "aten/src/ATen/ATenConfig.cmake.in": "cmake",
+       "*.py.in": "python",
+       "*.pyi.in": "python",
    },
    "java.autobuild.enabled": false,
    "java.compile.nullAnalysis.mode": "disabled",
+   "files.eol": "\n",
+   "files.insertFinalNewline": true,
+   "files.trimFinalNewlines": true,
+   "files.trimTrailingWhitespace": true,
+   "python.formatting.provider": "none",
+   "python.linting.enabled": true,
+   "python.linting.flake8Enabled": true,
}

janeyx99

Thanks for the change! Looks fine overall, just had some minor points :D

janeyx99 · 2023-02-28T22:05:54Z

tools/vscode_settings.py

+        current_settings = json.loads(current_settings_text)
+    except ValueError:  # json.JSONDecodeError is a subclass of ValueError
+        if HAS_JSON5:
+            raise SystemExit("Failed to parse .vscode/settings.json.")


Why SystemExit and not RuntimeError?

Since tools/vscode_settings.py is a script to run in the shell. It is not meant to be imported, so its functions will not be put in try-execpt blocks in other scripts.

Raising SystemExit is equivalent to printing an error message and exiting with code 1:

$ pip3 uninstall json5 $ ./tools/vscode_settings.py Failed to parse .vscode/settings.json. Maybe it contains comments or trailing commas. Try `pip install json5` to install an extended JSON parser. $ echo $? 1

Using RuntimeError:

$ pip3 uninstall json5 $ ./tools/vscode_settings.py Traceback (most recent call last): File "/home/PanXuehai/Projects/pytorch/./tools/vscode_settings.py", line 42, in main current_settings = json.loads(current_settings_text) File "/home/PanXuehai/Miniconda3/envs/pytorch-dev/lib/python3.10/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/home/PanXuehai/Miniconda3/envs/pytorch-dev/lib/python3.10/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/home/PanXuehai/Miniconda3/envs/pytorch-dev/lib/python3.10/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 5 column 5 (char 85) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/PanXuehai/Projects/pytorch/./tools/vscode_settings.py", line 65, in <module> main() File "/home/PanXuehai/Projects/pytorch/./tools/vscode_settings.py", line 46, in main raise RuntimeError( RuntimeError: Failed to parse .vscode/settings.json. Maybe it contains comments or trailing commas. Try `pip install json5` to install an extended JSON parser. $ echo $? 1

Ah, thank you for explaining!

tools/vscode_settings.py

XuehaiPan · 2023-03-01T15:50:02Z

@pytorchbot merge

pytorchmergebot · 2023-03-01T15:53:42Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…e annotated `NamedTuple` (#95267) Changes: - #95200 1. Recognize `.py.in` and `.pyi.in` files as Python in VS Code for a better development experience. 2. Fix deep setting merge in `tools/vscode_settings.py`. - => this PR: #95267 3. Use `Namedtuple` rather than `namedtuple + __annotations__` for `torch.nn.utils.rnn.PackedSequence_`: `namedtuple + __annotations__`: ```python PackedSequence_ = namedtuple('PackedSequence_', ['data', 'batch_sizes', 'sorted_indices', 'unsorted_indices']) # type annotation for PackedSequence_ to make it compatible with TorchScript PackedSequence_.__annotations__ = {'data': torch.Tensor, 'batch_sizes': torch.Tensor, 'sorted_indices': Optional[torch.Tensor], 'unsorted_indices': Optional[torch.Tensor]} ``` `Namedtuple`: Python 3.6+ ```python class PackedSequence_(NamedTuple): data: torch.Tensor batch_sizes: torch.Tensor sorted_indices: Optional[torch.Tensor] unsorted_indices: Optional[torch.Tensor] ``` - #95268 4. Sort import statements and remove unnecessary imports in `.pyi`, `.pyi.in` files. 5. Format `.pyi`, `.pyi.in` files and remove unnecessary ellipsis `...` in type stubs. Pull Request resolved: #95267 Approved by: https://github.com/janeyx99

) Changes: - #95200 1. Recognize `.py.in` and `.pyi.in` files as Python in VS Code for a better development experience. 2. Fix deep setting merge in `tools/vscode_settings.py`. - #95267 3. Use `Namedtuple` rather than `namedtuple + __annotations__` for `torch.nn.utils.rnn.PackedSequence_`: `namedtuple + __annotations__`: ```python PackedSequence_ = namedtuple('PackedSequence_', ['data', 'batch_sizes', 'sorted_indices', 'unsorted_indices']) # type annotation for PackedSequence_ to make it compatible with TorchScript PackedSequence_.__annotations__ = {'data': torch.Tensor, 'batch_sizes': torch.Tensor, 'sorted_indices': Optional[torch.Tensor], 'unsorted_indices': Optional[torch.Tensor]} ``` `Namedtuple`: Python 3.6+ ```python class PackedSequence_(NamedTuple): data: torch.Tensor batch_sizes: torch.Tensor sorted_indices: Optional[torch.Tensor] unsorted_indices: Optional[torch.Tensor] ``` - => this PR: #95268 4. Sort import statements and remove unnecessary imports in `.pyi`, `.pyi.in` files. 5. Format `.pyi`, `.pyi.in` files and remove unnecessary ellipsis `...` in type stubs. Pull Request resolved: #95268 Approved by: https://github.com/huydhn

…5200) Changes: - => this PR: #95200 1. Recognize `.py.in` and `.pyi.in` files as Python in VS Code for a better development experience. 2. Fix deep setting merge in `tools/vscode_settings.py`. - #95267 3. Use `Namedtuple` rather than `namedtuple + __annotations__` for `torch.nn.utils.rnn.PackedSequence_`: `namedtuple + __annotations__`: ```python PackedSequence_ = namedtuple('PackedSequence_', ['data', 'batch_sizes', 'sorted_indices', 'unsorted_indices']) # type annotation for PackedSequence_ to make it compatible with TorchScript PackedSequence_.__annotations__ = {'data': torch.Tensor, 'batch_sizes': torch.Tensor, 'sorted_indices': Optional[torch.Tensor], 'unsorted_indices': Optional[torch.Tensor]} ``` `Namedtuple`: Python 3.6+ ```python class PackedSequence_(NamedTuple): data: torch.Tensor batch_sizes: torch.Tensor sorted_indices: Optional[torch.Tensor] unsorted_indices: Optional[torch.Tensor] ``` - #95268 4. Sort import statements and remove unnecessary imports in `.pyi`, `.pyi.in` files. 5. Format `.pyi`, `.pyi.in` files and remove unnecessary ellipsis `...` in type stubs. Pull Request resolved: pytorch/pytorch#95200 Approved by: https://github.com/janeyx99

…Code (pytorch#95200)" This reverts commit a46e550.

XuehaiPan requested review from albanD and jbschlosser as code owners February 21, 2023 09:51

pytorch-bot bot added the release notes: distributed (c10d) release notes category label Feb 21, 2023

pytorchbot added the open source label Feb 21, 2023

XuehaiPan force-pushed the vscode-py-in branch 2 times, most recently from 1e997b5 to f10b578 Compare February 21, 2023 13:11

XuehaiPan requested review from mrshenli, zhaojuanmao, rohan-varma, H-Huang, awgu, kwen2501, wanchaol, fegin and janeyx99 as code owners February 21, 2023 13:11

XuehaiPan force-pushed the vscode-py-in branch from f10b578 to f2117c1 Compare February 21, 2023 13:15

XuehaiPan changed the title ~~Recognize .py.in and .pyi.in files as Python in VS Code~~ Update .pyi Python stub files and recognize .py.in and .pyi.in files as Python in VS Code Feb 21, 2023

XuehaiPan commented Feb 21, 2023

View reviewed changes

XuehaiPan force-pushed the vscode-py-in branch 4 times, most recently from fa34621 to 5013d55 Compare February 21, 2023 14:24

XuehaiPan changed the title ~~Update .pyi Python stub files and recognize .py.in and .pyi.in files as Python in VS Code~~ [1/3] Update .pyi Python stub files and recognize .py.in and .pyi.in files as Python in VS Code Feb 22, 2023

This was referenced Feb 22, 2023

[2/3] Update .pyi Python stub files: Prettify rnn.py by using type annotated NamedTuple #95267

Closed

[3/3] Update .pyi Python stub files and enable 'UFMT' linter #95268

Closed

Recognize .py.in and .pyi.in files as Python in VS Code

dcc1a6c

XuehaiPan force-pushed the vscode-py-in branch from 5013d55 to 7173060 Compare February 22, 2023 05:07

Fix tools/vscode_settings.py

e7ded55

XuehaiPan force-pushed the vscode-py-in branch from 7173060 to e7ded55 Compare February 22, 2023 05:45

bdhirsh added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Feb 22, 2023

janeyx99 reviewed Feb 28, 2023

View reviewed changes

XuehaiPan added 2 commits March 1, 2023 05:33

Apply suggestions from code review

1612c22

Merge branch 'master' into vscode-py-in

391b2f5

janeyx99 approved these changes Mar 1, 2023

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Mar 1, 2023

XuehaiPan changed the title ~~[1/3] Update .pyi Python stub files and recognize .py.in and .pyi.in files as Python in VS Code~~ [1/3] Recognize .py.in and .pyi.in files as Python in VS Code Mar 1, 2023

pytorch-bot bot added the topic: not user facing topic category label Mar 1, 2023

pytorchmergebot added the Merged label Mar 1, 2023

pytorchmergebot closed this in a46e550 Mar 1, 2023

XuehaiPan deleted the vscode-py-in branch March 2, 2023 05:49

pruthvistony added a commit to ROCm/pytorch that referenced this pull request May 2, 2023

Revert "[1/3] Recognize .py.in and .pyi.in files as Python in VS …

579cbcf

…Code (pytorch#95200)" This reverts commit a46e550.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[1/3] Recognize `.py.in` and `.pyi.in` files as Python in VS Code #95200

[1/3] Recognize `.py.in` and `.pyi.in` files as Python in VS Code #95200

Uh oh!

XuehaiPan commented Feb 21, 2023 •

edited

Loading

Uh oh!

pytorch-bot bot commented Feb 21, 2023 •

edited

Loading

Uh oh!

XuehaiPan Feb 21, 2023 •

edited

Loading

Uh oh!

albanD commented Feb 21, 2023

Uh oh!

XuehaiPan commented Feb 22, 2023 •

edited

Loading

Uh oh!

janeyx99 left a comment

Uh oh!

janeyx99 Feb 28, 2023

Uh oh!

XuehaiPan Mar 1, 2023 •

edited

Loading

Uh oh!

janeyx99 Mar 1, 2023

Uh oh!

Uh oh!

XuehaiPan commented Mar 1, 2023

Uh oh!

pytorchmergebot commented Mar 1, 2023

Uh oh!

Uh oh!

[1/3] Recognize .py.in and .pyi.in files as Python in VS Code #95200

[1/3] Recognize .py.in and .pyi.in files as Python in VS Code #95200

Uh oh!

Conversation

XuehaiPan commented Feb 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Feb 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/95200

✅ No Failures

Uh oh!

XuehaiPan Feb 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

albanD commented Feb 21, 2023

Uh oh!

XuehaiPan commented Feb 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janeyx99 left a comment

Choose a reason for hiding this comment

Uh oh!

janeyx99 Feb 28, 2023

Choose a reason for hiding this comment

Uh oh!

XuehaiPan Mar 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janeyx99 Mar 1, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

XuehaiPan commented Mar 1, 2023

Uh oh!

pytorchmergebot commented Mar 1, 2023

Merge started

Uh oh!

Uh oh!

[1/3] Recognize `.py.in` and `.pyi.in` files as Python in VS Code #95200

[1/3] Recognize `.py.in` and `.pyi.in` files as Python in VS Code #95200

XuehaiPan commented Feb 21, 2023 •

edited

Loading

pytorch-bot bot commented Feb 21, 2023 •

edited

Loading

XuehaiPan Feb 21, 2023 •

edited

Loading

XuehaiPan commented Feb 22, 2023 •

edited

Loading

XuehaiPan Mar 1, 2023 •

edited

Loading