Closed
Description
Describe the bug
[2023-09-02T23:10:26.261Z] Running ./computer_assisted_intervention/endoscopic_inbody_classification.ipynb
[2023-09-02T23:10:26.261Z] Checking PEP8 compliance...
[2023-09-02T23:10:26.823Z] Running notebook...
[2023-09-02T23:10:26.823Z] Before:
[2023-09-02T23:10:26.823Z] "max_epochs = 5\n",
[2023-09-02T23:10:27.079Z] After:
[2023-09-02T23:10:27.079Z] "max_epochs = 1\n",
[2023-09-02T23:10:33.620Z] MONAI version: 1.2.0+100.g50e66fa6
[2023-09-02T23:10:33.620Z] Numpy version: 1.22.2
[2023-09-02T23:10:33.620Z] Pytorch version: 2.1.0a0+29c30b1
[2023-09-02T23:10:33.620Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2023-09-02T23:10:33.620Z] MONAI rev id: 50e66fa646eec5cc4b84d0c0721bd2a5c5fc2e26
[2023-09-02T23:10:33.620Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2023-09-02T23:10:33.620Z]
[2023-09-02T23:10:33.620Z] Optional dependencies:
[2023-09-02T23:10:33.620Z] Pytorch Ignite version: 0.4.11
[2023-09-02T23:10:33.620Z] ITK version: 5.3.0
[2023-09-02T23:10:33.620Z] Nibabel version: 5.1.0
[2023-09-02T23:10:33.620Z] scikit-image version: 0.21.0
[2023-09-02T23:10:33.620Z] scipy version: 1.11.1
[2023-09-02T23:10:33.620Z] Pillow version: 9.2.0
[2023-09-02T23:10:33.620Z] Tensorboard version: 2.9.0
[2023-09-02T23:10:33.620Z] gdown version: 4.7.1
[2023-09-02T23:10:33.620Z] TorchVision version: 0.16.0a0
[2023-09-02T23:10:33.620Z] tqdm version: 4.65.0
[2023-09-02T23:10:33.620Z] lmdb version: 1.4.1
[2023-09-02T23:10:33.620Z] psutil version: 5.9.4
[2023-09-02T23:10:33.620Z] pandas version: 1.5.2
[2023-09-02T23:10:33.620Z] einops version: 0.6.1
[2023-09-02T23:10:33.620Z] transformers version: 4.21.3
[2023-09-02T23:10:33.620Z] mlflow version: 2.6.0
[2023-09-02T23:10:33.620Z] pynrrd version: 1.0.0
[2023-09-02T23:10:33.620Z] clearml version: 1.12.2
[2023-09-02T23:10:33.620Z]
[2023-09-02T23:10:33.620Z] For details about installing the optional dependencies, please visit:
[2023-09-02T23:10:33.620Z] https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2023-09-02T23:10:33.620Z]
[2023-09-02T23:10:36.884Z] papermill --progress-bar -k python3
[2023-09-02T23:10:36.884Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2023-09-02T23:10:36.884Z] warnings.warn(
[2023-09-02T23:11:05.834Z]
Executing: 0%| | 0/17 [00:00<?, ?cell/s]
Executing: 6%|▌ | 1/17 [00:01<00:20, 1.31s/cell]
Executing: 18%|█▊ | 3/17 [00:10<00:52, 3.72s/cell]
Executing: 29%|██▉ | 5/17 [00:16<00:40, 3.40s/cell]
Executing: 53%|█████▎ | 9/17 [00:17<00:12, 1.52s/cell]
Executing: 65%|██████▍ | 11/17 [00:18<00:07, 1.18s/cell]
Executing: 76%|███████▋ | 13/17 [00:18<00:03, 1.12cell/s]
Executing: 88%|████████▊ | 15/17 [00:25<00:03, 1.69s/cell]
Executing: 88%|████████▊ | 15/17 [00:28<00:03, 1.90s/cell]
[2023-09-02T23:11:05.834Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2023-09-02T23:11:05.834Z] warnings.warn(
[2023-09-02T23:11:05.834Z] Traceback (most recent call last):
[2023-09-02T23:11:05.834Z] File "/usr/local/bin/papermill", line 8, in <module>
[2023-09-02T23:11:05.834Z] sys.exit(papermill())
[2023-09-02T23:11:05.834Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
[2023-09-02T23:11:05.834Z] return self.main(*args, **kwargs)
[2023-09-02T23:11:05.834Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
[2023-09-02T23:11:05.834Z] rv = self.invoke(ctx)
[2023-09-02T23:11:05.834Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
[2023-09-02T23:11:05.834Z] return ctx.invoke(self.callback, **ctx.params)
[2023-09-02T23:11:05.834Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
[2023-09-02T23:11:05.834Z] return __callback(*args, **kwargs)
[2023-09-02T23:11:05.834Z] File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func
[2023-09-02T23:11:05.834Z] return f(get_current_context(), *args, **kwargs)
[2023-09-02T23:11:05.834Z] File "/usr/local/lib/python3.10/dist-packages/papermill/cli.py", line 250, in papermill
[2023-09-02T23:11:05.834Z] execute_notebook(
[2023-09-02T23:11:05.834Z] File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 128, in execute_notebook
[2023-09-02T23:11:05.834Z] raise_for_execution_errors(nb, output_path)
[2023-09-02T23:11:05.834Z] File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 232, in raise_for_execution_errors
[2023-09-02T23:11:05.834Z] raise error
[2023-09-02T23:11:05.834Z] papermill.exceptions.PapermillExecutionError:
[2023-09-02T23:11:05.834Z] ---------------------------------------------------------------------------
[2023-09-02T23:11:05.834Z] Exception encountered at "In [7]":
[2023-09-02T23:11:05.834Z] ---------------------------------------------------------------------------
[2023-09-02T23:11:05.834Z] TypeError Traceback (most recent call last)
[2023-09-02T23:11:05.834Z] Cell In[7], line 5
[2023-09-02T23:11:05.834Z] 1 max_epochs = 1
[2023-09-02T23:11:05.834Z] 3 pretrained_model = monai.bundle.load(name="endoscopic_inbody_classification", bundle_dir="./")
[2023-09-02T23:11:05.834Z] ----> 5 model.load_state_dict(pretrained_model)
[2023-09-02T23:11:05.834Z] 7 model.train()
[2023-09-02T23:11:05.834Z] 8 losses = []
[2023-09-02T23:11:05.834Z]
[2023-09-02T23:11:05.834Z] File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:2042, in Module.load_state_dict(self, state_dict, strict, assign)
[2023-09-02T23:11:05.834Z] 2008 r"""Copies parameters and buffers from :attr:`state_dict` into
[2023-09-02T23:11:05.834Z] 2009 this module and its descendants. If :attr:`strict` is ``True``, then
[2023-09-02T23:11:05.834Z] 2010 the keys of :attr:`state_dict` must exactly match the keys returned
[2023-09-02T23:11:05.834Z] (...)
[2023-09-02T23:11:05.834Z] 2039 ``RuntimeError``.
[2023-09-02T23:11:05.834Z] 2040 """
[2023-09-02T23:11:05.834Z] 2041 if not isinstance(state_dict, Mapping):
[2023-09-02T23:11:05.834Z] -> 2042 raise TypeError("Expected state_dict to be dict-like, got {}.".format(type(state_dict)))
[2023-09-02T23:11:05.834Z] 2044 missing_keys: List[str] = []
[2023-09-02T23:11:05.834Z] 2045 unexpected_keys: List[str] = []
[2023-09-02T23:11:05.834Z]
[2023-09-02T23:11:05.834Z] TypeError: Expected state_dict to be dict-like, got <class 'monai.networks.nets.senet.SEResNet50'>.
[2023-09-02T22:44:42.683Z] Running ./experiment_management/bundle_integrate_mlflow.ipynb
[2023-09-02T22:44:42.683Z] Checking PEP8 compliance...
[2023-09-02T22:44:42.683Z] Running notebook...
[2023-09-02T22:44:42.683Z] Before:
[2023-09-02T22:44:42.683Z] "workflow.max_epochs = 10\n",
[2023-09-02T22:44:42.941Z] After:
[2023-09-02T22:44:42.941Z] "workflow.max_epochs = 1\n",
[2023-09-02T22:44:51.045Z] MONAI version: 1.2.0+100.g50e66fa6
[2023-09-02T22:44:51.045Z] Numpy version: 1.22.2
[2023-09-02T22:44:51.045Z] Pytorch version: 2.1.0a0+29c30b1
[2023-09-02T22:44:51.045Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2023-09-02T22:44:51.045Z] MONAI rev id: 50e66fa646eec5cc4b84d0c0721bd2a5c5fc2e26
[2023-09-02T22:44:51.045Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2023-09-02T22:44:51.045Z]
[2023-09-02T22:44:51.045Z] Optional dependencies:
[2023-09-02T22:44:51.045Z] Pytorch Ignite version: 0.4.11
[2023-09-02T22:44:51.045Z] ITK version: 5.3.0
[2023-09-02T22:44:51.045Z] Nibabel version: 5.1.0
[2023-09-02T22:44:51.045Z] scikit-image version: 0.21.0
[2023-09-02T22:44:51.045Z] scipy version: 1.11.1
[2023-09-02T22:44:51.045Z] Pillow version: 9.2.0
[2023-09-02T22:44:51.045Z] Tensorboard version: 2.9.0
[2023-09-02T22:44:51.045Z] gdown version: 4.7.1
[2023-09-02T22:44:51.045Z] TorchVision version: 0.16.0a0
[2023-09-02T22:44:51.045Z] tqdm version: 4.65.0
[2023-09-02T22:44:51.045Z] lmdb version: 1.4.1
[2023-09-02T22:44:51.045Z] psutil version: 5.9.4
[2023-09-02T22:44:51.045Z] pandas version: 1.5.2
[2023-09-02T22:44:51.045Z] einops version: 0.6.1
[2023-09-02T22:44:51.045Z] transformers version: 4.21.3
[2023-09-02T22:44:51.045Z] mlflow version: 2.6.0
[2023-09-02T22:44:51.045Z] pynrrd version: 1.0.0
[2023-09-02T22:44:51.045Z] clearml version: 1.12.2
[2023-09-02T22:44:51.045Z]
[2023-09-02T22:44:51.045Z] For details about installing the optional dependencies, please visit:
[2023-09-02T22:44:51.045Z] https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2023-09-02T22:44:51.045Z]
[2023-09-02T22:44:54.324Z] papermill --progress-bar -k python3
[2023-09-02T22:44:54.887Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2023-09-02T22:44:54.887Z] warnings.warn(
[2023-09-02T22:47:07.397Z]
Executing: 0%| | 0/27 [00:00<?, ?cell/s]
Executing: 4%|▎ | 1/27 [00:01<00:33, 1.27s/cell]
Executing: 11%|█ | 3/27 [00:09<01:19, 3.31s/cell]
Executing: 19%|█▊ | 5/27 [00:15<01:09, 3.15s/cell]
Executing: 33%|███▎ | 9/27 [01:42<04:16, 14.23s/cell]
Executing: 44%|████▍ | 12/27 [01:45<02:15, 9.03s/cell]
Executing: 56%|█████▌ | 15/27 [01:53<01:21, 6.82s/cell]
Executing: 67%|██████▋ | 18/27 [02:01<00:48, 5.42s/cell]
Executing: 81%|████████▏ | 22/27 [02:09<00:20, 4.08s/cell]
Executing: 96%|█████████▋| 26/27 [02:10<00:02, 2.66s/cell]
Executing: 96%|█████████▋| 26/27 [02:12<00:05, 5.10s/cell]
[2023-09-02T22:47:07.397Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2023-09-02T22:47:07.397Z] warnings.warn(
[2023-09-02T22:47:07.655Z] Traceback (most recent call last):
[2023-09-02T22:47:07.655Z] File "/usr/local/bin/papermill", line 8, in <module>
[2023-09-02T22:47:07.655Z] sys.exit(papermill())
[2023-09-02T22:47:07.655Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
[2023-09-02T22:47:07.655Z] return self.main(*args, **kwargs)
[2023-09-02T22:47:07.655Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
[2023-09-02T22:47:07.655Z] rv = self.invoke(ctx)
[2023-09-02T22:47:07.655Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
[2023-09-02T22:47:07.655Z] return ctx.invoke(self.callback, **ctx.params)
[2023-09-02T22:47:07.655Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
[2023-09-02T22:47:07.655Z] return __callback(*args, **kwargs)
[2023-09-02T22:47:07.655Z] File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func
[2023-09-02T22:47:07.655Z] return f(get_current_context(), *args, **kwargs)
[2023-09-02T22:47:07.655Z] File "/usr/local/lib/python3.10/dist-packages/papermill/cli.py", line 250, in papermill
[2023-09-02T22:47:07.655Z] execute_notebook(
[2023-09-02T22:47:07.655Z] File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 128, in execute_notebook
[2023-09-02T22:47:07.655Z] raise_for_execution_errors(nb, output_path)
[2023-09-02T22:47:07.655Z] File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 232, in raise_for_execution_errors
[2023-09-02T22:47:07.655Z] raise error
[2023-09-02T22:47:07.655Z] papermill.exceptions.PapermillExecutionError:
[2023-09-02T22:47:07.655Z] ---------------------------------------------------------------------------
[2023-09-02T22:47:07.655Z] Exception encountered at "In [9]":
[2023-09-02T22:47:07.655Z] ---------------------------------------------------------------------------
[2023-09-02T22:47:07.655Z] AttributeError Traceback (most recent call last)
[2023-09-02T22:47:07.655Z] Cell In[9], line 31
[2023-09-02T22:47:07.655Z] 29 workflow.max_epochs = 1
[2023-09-02T22:47:07.655Z] 30 workflow.initialize()
[2023-09-02T22:47:07.655Z] ---> 31 train_mlflow_handler.attach(workflow.trainer)
[2023-09-02T22:47:07.655Z] 32 val_mlflow_handler.attach(workflow.evaluator)
[2023-09-02T22:47:07.655Z] 33 workflow.run()
[2023-09-02T22:47:07.655Z]
[2023-09-02T22:47:07.655Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/bundle/workflows.py:126, in BundleWorkflow.__getattr__(self, name)
[2023-09-02T22:47:07.655Z] 124 return self._get_property(name=name, property=self.properties[name])
[2023-09-02T22:47:07.655Z] 125 else:
[2023-09-02T22:47:07.655Z] --> 126 return self.__getattribute__(name)
[2023-09-02T22:47:07.655Z]
[2023-09-02T22:47:07.655Z] AttributeError: 'ConfigWorkflow' object has no attribute 'trainer'