jsonserver high RAM usage

[robmcmullen]

Expected Behaviour

jsonserver using a relatively static amount of RAM

Actual Behaviour

jsonserver process increases size, seemingly without bounds until it swaps everything out of memory except itself.

Steps to Reproduce

As noted in #318, and by @qocu it's not clear exactly what is causing it, but it seems to be related to large projects or multiple Sublime windows open in a project

ST3, Anaconda and OS versions

ST3126, Anaconda 2.1.18, linux 4.1.12 kernel

ST3 Console Logs

Nothing useful I expect, but here: st3.txt

Anaconda's JsonServer Logs

I've not had it fail again since this jsonserver log; will update when it happens again. anaconda-traceback.txt

[robmcmullen]

Forgot to mention I'm using a 2.7.10 interpreter in a virtualenv

[qocu]

Python: 3.4.3 in virtualenv ( multiple virtualenvs in multiple sublime windows )
OS: Linux kernel 4.9.6, Gentoo
Projects have a lot of different files aside from python files, html, css, js etc ( django projects )
Only way to reproduce I've encountered is when this happens one. When I kill jsonserver without restarting sublime problems occurs again after a while.

[robmcmullen]

I'm on Gentoo also; probably not a factor, but still interesting.

I'm using two really big libraries in the virtualenv, wxPython (related to davidhalter/jedi#805 probably) and the Enthought Traits/Pyface libraries. There's also a fair amount of Cython code, but I think Jedi skips over that?

[DamnWidget]

I will try to reproduce it, things go rogue when two windows are open using the same project is probably logical on Linux as there is only one Unix Domain Socket and two windows trying to start anaconda server over that socket, with only one window is weird.

What I can see in the logs is that Jedi breaks.

[DamnWidget]

Ok I did several crazy things to try to reproduce the logs that @robmcmullen pasted but I was not able to make Jedi go rogue I just exhausted my computer RAM and had to restart.

To do so, I had to open 7 windows in a project that was using wxPython, each instance of Jedi that needs to complete wxPython uses 6Gb of memory (I have 32GB on this computer)

Jedi seems to use that huge amount of memory always that imports the xwPython module as use iPython demonstrates
22233 damnwid+ 20 0 6457.0m 6.005g 0.0 19.1 0:25.90 S ipython2

We have a problem here because the last version of Jedi that is on master, fixed this problem, this is the comparative between the current jedi version bundled into anaconda and the latest one:

Anaconda's bundled version:
17880 damnwidge 20 0 6399M 6156M 8300 S 0.0 19.2 0:00.00 /usr/bin/python2 /usr/bin/ipython2

Latest Jedi version:
15365 damnwidge 20 0 574M 186M 25200 S 0.0 0.6 0:00.00 /usr/bin/python2 /usr/bin/ipython2

As you can see, is much better but we have a problem with the latest Jedi version but that version caused some performance issues in OSX and I had to rollback it #632

So what we have here is a big problem to me because I do not have the time to hunt down the problem in OSX and update Jedi, apart from that latest version of Jedi API is not compatible with anaconda and I had to modify anaconda to make it work with it so it is not an in-drop replacement and probably that could be causing some issue.

I will update this as soon as I have anything to update.

P.S: as a suggestion, try to do not open more than one window into the project and if you do use another Python interpreter for the second one, one interpreter that does not see the library that is using too many memory

[robmcmullen]

Thanks for taking a look at it. Sounds like we can't just plop in the new dev version of Jedi because of the API changes. I'd be willing to try to backport the fix from the current Jedi to the version that is compatible with Anaconda, but I can't find the commit in Jedi that fixes it. Any ideas which commit might have been the resolution? Or was it a series of changes that would be difficult to backport?

[DamnWidget]

I think is easier if you just use the jedi_bump anaconda branch

[robmcmullen]

At the jedi_bump branch, it's made a huge difference so far. jsonserver seems to be holding in the 300-400MB range.

[DamnWidget]

@qocu did you try the jedi_bump branch? It improved your experience?

[robmcmullen]

The same python process has been running for these past 4 days and it has only grown slightly to 500MB. I'd say that it fixes my issue.

[qocu]

@DamnWidget haven't tried it yet, I'll be working on large projects again ( on small I haven't noticed problems even on master branch ) in couple of days so I will post observations.

[robmcmullen]

Aaaaand, I jinxed myself. The jsonserver process is currently spinning and growing, but pretty slowly compared to last time. In 4 minutes, it's only at 1.2GB. I couldn't get jsonserver to run in debug mode using jedi_bump, for some reason it would never talk to sublime. I'll try again.

[robmcmullen]

Got jsonserver running in debug mode. I'll post a followup if it starts spinning again in a few days.

[robmcmullen]

Could it be related to different projects using the same virtualenv? I have two related projects that I was using the same virtualenv to edit and test, both projects were open and both were getting linted automatically.

In debug mode, you specify a single project and it doesn't appear to do any linting or anything of other projects that are also open using the same virtualenv. I'll try a second debug jsonserver for the other project and see what happens.