Wierd Bug - No idea what happened

[tchaton]

terminate called after throwing an instance of 'c10::Error'
  what():  owning_ptr == NullType::singleton() || owning_ptr->refcount_.load() > 0 ASSERT FAILED at /pytorch/c10/util/intrusive_ptr.h:350, please report a bug to PyTorch. intrusive_ptr: Can only intrusive_ptr::reclaim() owning pointers that were created using intrusive_ptr::release(). (reclaim at /pytorch/c10/util/intrusive_ptr.h:350)
frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7f71d2915021 in /home/tchaton/virtualenvs/labelbox/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7f71d29148ea in /home/tchaton/virtualenvs/labelbox/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #2: THStorage_free + 0xca (0x7f71d36bc0fa in /home/tchaton/virtualenvs/labelbox/lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #3: <unknown function> + 0x4aca27 (0x7f71dcf4fa27 in /home/tchaton/virtualenvs/labelbox/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: /home/tchaton/virtualenvs/labelbox/bin/python() [0x4c12c4]
frame #5: /home/tchaton/virtualenvs/labelbox/bin/python() [0x49b567]
frame #6: /home/tchaton/virtualenvs/labelbox/bin/python() [0x49c450]
frame #7: /home/tchaton/virtualenvs/labelbox/bin/python() [0x5c9024]
frame #8: /home/tchaton/virtualenvs/labelbox/bin/python() [0x483e69]
frame #9: /home/tchaton/virtualenvs/labelbox/bin/python() [0x435cc7]
frame #10: /home/tchaton/virtualenvs/labelbox/bin/python() [0x435cd7]
frame #11: /home/tchaton/virtualenvs/labelbox/bin/python() [0x435cd7]
frame #12: _PyEval_EvalFrameDefault + 0x4f8c (0x5465ec in /home/tchaton/virtualenvs/labelbox/bin/python)
frame #13: /home/tchaton/virtualenvs/labelbox/bin/python() [0x53f8d1]
frame #14: /home/tchaton/virtualenvs/labelbox/bin/python() [0x540781]
frame #15: _PyEval_EvalFrameDefault + 0x102d (0x54268d in /home/tchaton/virtualenvs/labelbox/bin/python)
frame #16: /home/tchaton/virtualenvs/labelbox/bin/python() [0x53f8d1]
frame #17: /home/tchaton/virtualenvs/labelbox/bin/python() [0x540781]
frame #18: _PyEval_EvalFrameDefault + 0x102d (0x54268d in /home/tchaton/virtualenvs/labelbox/bin/python)
frame #19: /home/tchaton/virtualenvs/labelbox/bin/python() [0x53f8d1]
frame #20: /home/tchaton/virtualenvs/labelbox/bin/python() [0x540781]
frame #21: _PyEval_EvalFrameDefault + 0x102d (0x54268d in /home/tchaton/virtualenvs/labelbox/bin/python)
frame #22: /home/tchaton/virtualenvs/labelbox/bin/python() [0x53f8d1]
frame #23: /home/tchaton/virtualenvs/labelbox/bin/python() [0x540781]
frame #24: _PyEval_EvalFrameDefault + 0x102d (0x54268d in /home/tchaton/virtualenvs/labelbox/bin/python)
frame #25: /home/tchaton/virtualenvs/labelbox/bin/python() [0x53f8d1]
frame #26: /home/tchaton/virtualenvs/labelbox/bin/python() [0x540781]
frame #27: _PyEval_EvalFrameDefault + 0x102d (0x54268d in /home/tchaton/virtualenvs/labelbox/bin/python)
frame #28: /home/tchaton/virtualenvs/labelbox/bin/python() [0x53f8d1]
frame #29: /home/tchaton/virtualenvs/labelbox/bin/python() [0x540781]
frame #30: _PyEval_EvalFrameDefault + 0x102d (0x54268d in /home/tchaton/virtualenvs/labelbox/bin/python)
frame #31: /home/tchaton/virtualenvs/labelbox/bin/python() [0x53f8d1]
frame #32: /home/tchaton/virtualenvs/labelbox/bin/python() [0x540781]
frame #33: _PyEval_EvalFrameDefault + 0x102d (0x54268d in /home/tchaton/virtualenvs/labelbox/bin/python)
frame #34: /home/tchaton/virtualenvs/labelbox/bin/python() [0x540275]
frame #35: PyEval_EvalCode + 0x23 (0x541103 in /home/tchaton/virtualenvs/labelbox/bin/python)
frame #36: PyRun_FileExFlags + 0x16f (0x42777f in /home/tchaton/virtualenvs/labelbox/bin/python)
frame #37: PyRun_SimpleFileExFlags + 0xec (0x4279ac in /home/tchaton/virtualenvs/labelbox/bin/python)
frame #38: Py_Main + 0xd85 (0x43be55 in /home/tchaton/virtualenvs/labelbox/bin/python)
frame #39: main + 0x162 (0x41dd42 in /home/tchaton/virtualenvs/labelbox/bin/python)
frame #40: __libc_start_main + 0xf0 (0x7f71eee10830 in /lib/x86_64-linux-gnu/libc.so.6)
frame #41: _start + 0x29 (0x41de19 in /home/tchaton/virtualenvs/labelbox/bin/python)

Aborted (core dumped)

[albanD]

Hi.

Could you give more context how you installed pytorch and what were you doing when you get this error?
In particular if you can give a small code sample that reproduce the crast, that would be very helpful !

[kparaju]

I'll update this post as I learn more.

Seeing the same error. Installed via conda. I am loading a serialized model. I also saw the following error before:

Traceback (most recent call last):
  File "/opt/conda/envs/exp_ta_rec_rfy_v4_serve/lib/python3.5/site-packages/gunicorn-19.1.0-py3.5.egg/gunicorn/arbiter.py", line 507, in spawn_worker
    worker.init_process()
  File "/opt/conda/envs/exp_ta_rec_rfy_v4_serve/lib/python3.5/site-packages/gunicorn-19.1.0-py3.5.egg/gunicorn/workers/base.py", line 114, in init_process
    self.wsgi = self.app.wsgi()
  File "/opt/conda/envs/exp_ta_rec_rfy_v4_serve/lib/python3.5/site-packages/gunicorn-19.1.0-py3.5.egg/gunicorn/app/base.py", line 66, in wsgi
    self.callable = self.load()
  File "/opt/conda/envs/exp_ta_rec_rfy_v4_serve/lib/python3.5/site-packages/gunicorn-19.1.0-py3.5.egg/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/opt/conda/envs/exp_ta_rec_rfy_v4_serve/lib/python3.5/site-packages/gunicorn-19.1.0-py3.5.egg/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/opt/conda/envs/exp_ta_rec_rfy_v4_serve/lib/python3.5/site-packages/gunicorn-19.1.0-py3.5.egg/gunicorn/util.py", line 356, in import_app
    __import__(module)
  File "/home/vrdsuser/exp_ta_rec_rfy_v4/exp_ta_rec_rfy_v4/serve/app.py", line 56, in <module>
    recommender = initialize()
  File "/home/vrdsuser/exp_ta_rec_rfy_v4/exp_ta_rec_rfy_v4/serve/app.py", line 49, in initialize
    model = runner.load(runner.name + '_' + model_name)
  File "/opt/conda/envs/exp_ta_rec_rfy_v4_serve/lib/python3.5/site-packages/internaltripadvisorlib/dsprunner.py", line 941, in load
    ret = persist_class().load_from_runner(self, name, metadata_dict)
  File "/opt/conda/envs/exp_ta_rec_rfy_v4_serve/lib/python3.5/site-packages/internaltripadvisorlib/dsprunner.py", line 269, in load_from_runner
    tr.add_model(model_name, dsp_runner.load_model(model_name))
  File "/opt/conda/envs/exp_ta_rec_rfy_v4_serve/lib/python3.5/site-packages/internaltripadvisorlib/dsprunner.py", line 854, in load_model
    directory=self._get_directory('artifact'))
  File "/opt/conda/envs/exp_ta_rec_rfy_v4_serve/lib/python3.5/site-packages/exp_rfycommon/model_trainer.py", line 216, in load_model
    pytorch_model = torch.load(model_path, map_location='cpu')
  File "/opt/conda/envs/exp_ta_rec_rfy_v4_serve/lib/python3.5/site-packages/torch/serialization.py", line 368, in load
    return _load(f, map_location, pickle_module)
  File "/opt/conda/envs/exp_ta_rec_rfy_v4_serve/lib/python3.5/site-packages/torch/serialization.py", line 549, in _load
    deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
RuntimeError: read(): fd 16 failed with Bad address

...

terminate called after throwing an instance of 'c10::Error'
  what():  owning_ptr == NullType::singleton() || owning_ptr->refcount_.load() > 0 ASSERT FAILED at /opt/conda/conda-bld/pytorch_1549624239588/work/c10/util/intrusive_ptr.h:350, please report a bug to PyTorch. intrusive_ptr: Can only intrusive_ptr::reclaim() owning pointers that were created using intrusive_ptr::release(). (reclaim at /opt/conda/conda-bld/pytorch_1549624239588/work/c10/util/intrusive_ptr.h:350)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x45 (0x7f097da08cf5 in /opt/conda/envs/exp_ta_rec_rfy_v4_serve/lib/python3.5/site-packages/torch/lib/libc10.so)
frame #1: THStorage_free + 0xca (0x7f0945dcc8ea in /opt/conda/envs/exp_ta_rec_rfy_v4_serve/lib/python3.5/site-packages/torch/lib/libcaffe2.so)
frame #2: <unknown function> + 0x12c12d (0x7f097dd4612d in /opt/conda/envs/exp_ta_rec_rfy_v4_serve/lib/python3.5/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #30: main + 0x15d (0x400add in /opt/conda/envs/exp_ta_rec_rfy_v4_serve/bin/python3)
frame #31: __libc_start_main + 0xf1 (0x7f09af1242e1 in /lib/x86_64-linux-gnu/libc.so.6)
frame #32: /opt/conda/envs/exp_ta_rec_rfy_v4_serve/bin/python3() [0x4008b9]

Here is the conda env export:

vrdsuser@exp-ta-rfy-service-6b99cdb56-vvwb4:~/exp_ta_rec_rfy_v4$ conda env export -n exp_ta_rec_rfy_v4_serve
name: exp_ta_rec_rfy_v4_serve
channels:
  - pytorch
  - conda-forge
  - defaults
dependencies:
  - blas=1.1=openblas
  - ca-certificates=2019.3.9=hecc5488_0
  - certifi=2018.8.24=py35_1001
  - cffi=1.11.5=py35h5e8e0c9_1
  - click=7.0=py_0
  - flask=1.0.2=py_2
  - freetds=1.00.44=1
  - itsdangerous=1.1.0=py_0
  - jinja2=2.10.1=py_0
  - libffi=3.2.1=he1b5a44_1006
  - libgfortran=3.0.0=1
  - markupsafe=1.0=py35h470a237_1
  - ncurses=5.9=10
  - ninja=1.9.0=h6bb024c_0
  - numpy=1.11.3=py35_blas_openblashd3ea46f_205
  - openblas=0.2.20=8
  - openssl=1.0.2r=h14c3975_0
  - pastedeploy=1.5.2=py_0
  - pip=9.0.3=py35_0
  - pycparser=2.19=py_0
  - pymssql=2.1.3.post16=py35_0
  - readline=6.2=0
  - scipy=1.1.0=py35_blas_openblash7943236_201
  - setuptools=40.4.3=py35_0
  - sqlite=3.19.3=1
  - tk=8.5.19=2
  - werkzeug=0.15.4=py_0
  - wheel=0.32.0=py35_1000
  - xz=5.0.5=1
  - zlib=1.2.11=h14c3975_1004
  - cudatoolkit=10.0.130=0
  - gunicorn=19.1.0=py35_0
  - intel-openmp=2019.4=243
  - libgcc-ng=9.1.0=hdf63c60_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - mkl=2019.4=243
  - python=3.5.0=1
  - pytorch=1.0.1=py3.5_cuda10.0.130_cudnn7.4.2_2
  - pip:
    - asn1crypto==0.24.0
    - bcrypt==3.1.6
    - bunch==1.0.1
    - cachetools==3.1.1
    - chardet==3.0.4
    - cryptography==2.7
    - exp-rfycommon==1.0
    - exp-ta-rec-rfy-v4==1.0
    - google-api-core==1.11.1
    - google-auth==1.6.3
    - google-auth-oauthlib==0.4.0
    - google-cloud-bigquery==1.11.2
    - google-cloud-core==0.29.1
    - google-cloud-storage==1.15.0
    - google-resumable-media==0.3.2
    - googleapis-common-protos==1.6.0
    - idna==2.8
    - oauthlib==3.0.1
    - pandas==0.24.0
    - pandas-gbq==0.3.0
    - paramiko==2.5.0
    - protobuf==3.8.0
    - psutil==5.4.3
    - pyasn1==0.4.5
    - pyasn1-modules==0.2.5
    - pymysql==0.8.0
    - pynacl==1.3.0
    - pysftp==0.2.9
    - python-dateutil==2.8.0
    - pytz==2019.1
    - pyyaml==3.12
    - requests==2.21.0
    - requests-oauthlib==1.2.0
    - rsa==4.0
    - scikit-learn==0.20.3
    - six==1.12.0
    - sqlalchemy==1.2.4
    - torch==1.0.1.post2
    - urllib3==1.24.3
    - internaltripadvisorlib==6.2.5
prefix: /opt/conda/envs/exp_ta_rec_rfy_v4_serve

Looking at one of the core dumps:

(base) root@exp-ta-rfy-service-6b99cdb56-vvwb4:/home/vrdsuser/exp_ta_rec_rfy_v4# gdb /opt/conda/envs/exp_ta_rec_rfy_v4_serve/bin/python3 core.2053
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/conda/envs/exp_ta_rec_rfy_v4_serve/bin/python3...done.

warning: core file may not match specified executable file.
[New LWP 2053]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Core was generated by `/opt/conda/envs/exp_ta_rec_rfy_v4_serve/bin/python3 /opt/conda/envs/exp_ta_rec_'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

[pietern]

@kparaju @tchaton Which serialized model are you loading? Are you using the same one?

I'm guessing the serialized model is malformed, or otherwise causes some tensors to be partially loaded or initialized, which in turn causes the assertion failure when the program terminates. That said, even for malformed serialized models, it should error gracefully, not terminate with an assert.

[knsong]

@tchaton Did you solve the problem?

[tchaton]

@knsong, I never figure out what really happened. I just think it could be due to the process being interrupted during writing the checkpoints.

[albanD]

Will close as the error does not seem to happen anymore based on @tchaton's comment.
Feel free to reopen if you see the error and have a reproduction.

[kontrabas380]

@albanD @knsong, I'm having similar issue to @tchaton:
I'm running my script in docker, and on few machines it works properly, but when I try to load my model on specific machine, I've got the same error as @kparaju which looks like:

[2020-02-24 15:27:38,215: ERROR/ForkPoolWorker-76] Task fastai-models[c40b5516-3461-484f-b9cf-c6caf9608f0a] raised unexpected: RuntimeError('read(): fd 33 failed with Bad address',) Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 385, in trace_task R = retval = fun(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 648, in __protected_call__ return self.run(*args, **kwargs) File "/code/tasks/async_tasks.py", line 10, in fastai-models job = Job(job_id) File "/code/internal_representation/job.py", line 17, in __init__ self.model_configs_dict = self.create_model_configs_from_data() File "/code/internal_representation/job.py", line 31, in create_model_configs_from_data model_configs_dict[service_name] = ModelConfig(model_config_data) File "/code/internal_representation/model_config.py", line 7, in __init__ self.model_groups_list = self.create_model_groups_from_config_data() File "/code/internal_representation/model_config.py", line 12, in create_model_groups_from_config_data model_group = ModelGroup(model_group_config) File "/code/internal_representation/model_group.py", line 8, in __init__ self.model_group = self.create_model_group() File "/code/internal_representation/model_group.py", line 17, in create_model_group model = Model(model_config).model File "/code/internal_representation/moderator.py", line 12, in __init__ self.model = self.create_model() File "/code/internal_representation/moderator.py", line 43, in create_model model = {'model': AiModel.get_instance(model_name=self.model_config['settings']['model_name']), File "/code/mixins/singleton_mixin.py", line 13, in get_instance cls.__singleton_instance = cls(*args, **kwargs) File "/code/model/ai_modelpy", line 15, in __init__ self.gossip_model = load_learner('model/models_production', 'else-exported.pkl') File "/usr/local/lib/python3.6/site-packages/fastai/basic_train.py", line 616, in load_learner state = torch.load(source, map_location='cpu') if defaults.device == torch.device('cpu') else torch.load(source) File "/usr/local/lib/python3.6/site-packages/torch/serialization.py", line 426, in load return _load(f, map_location, pickle_module, **pickle_load_args) File "/usr/local/lib/python3.6/site-packages/torch/serialization.py", line 620, in _load deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly) RuntimeError: read(): fd 33 failed with Bad address

Do you know what could be the problem?

Best regards

[albanD]

The errors seems to point to a bad file. Maybe you're moving files around while they're being used by docker? Or they are corrupted?

Without more information, it will be hard to help you I'm afraid.

[kontrabas380]

@albanD, it is correctly working on my machine with the same docker, so files are not corrupted. And model file is inside the docker so I'm not moving them around.

What information should I prepare?

[albanD]

You should get a script to reproduce this issue (30/40 lines).
Then open a new issue with the script + all the information requested in the bug-report template when you create an issue.
Thanks !