[kdevelop] [Bug 379551] New: Sudden huge memory leaks when using KDevelop for an extended amount of time

[Manvydas Šliamka]

https://bugs.kde.org/show_bug.cgi?id=379551

Bug ID: 379551
Summary: Sudden huge memory leaks when using KDevelop for an
extended amount of time
Product: kdevelop
Version: 5.1.0
Platform: Appimage
OS: Linux
Status: UNCONFIRMED
Severity: normal
Priority: NOR
Component: general
Assignee: kdevelop-bugs-***@kde.org
Reporter: ***@gmail.com
Target Milestone: ---

I'm using a KDevelop 5.1.0 Appimage on Ubuntu GNOME 17.04 on a fairly low RAM
(4 GiB) machine.

After using it for 2-3 hours, sometimes longer, I notice a severe and sudden
system slowdown. If I open htop or System Monitor, they show that KDevelop is
using several GiBs of RAM and that my pagefile is filling rapidly.

I'm almost sure that the slowdown only starts when the background parser is
running, however, I may be wrong.

Also, I've only started using KDevelop a week ago (I love it :D) and I've not
yet customized anything. I'm certain I am running on default settings.

One last thing. I thought that my project may be at fault, so I tried to play
around with considerably bigger codebases of several CMake based open source
projects (OpenCV and Assimp). The exact same thing happened after roughly the
same amount of time.

This is definitely not a critical bug, however, being forced to close (or even
kill) and restart KDevelop every 2-3 hours is definitely annoying.

[Milian Wolff]

https://bugs.kde.org/show_bug.cgi?id=379551

Milian Wolff <***@milianw.de> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|UNCONFIRMED |NEEDSINFO
Resolution|--- |WAITINGFORINFO

--- Comment #1 from Milian Wolff <***@milianw.de> ---
this sounds odd - do you have many tabs open?

in general, how do you track memory consumption? can you try a heap profiler
like heaptrack to see what happens and what consumes memory?

does anything happen when you do this once RSS memory consumption is reported
as high:

gdb --pid $(pidof kdevelop)
...
(gdb) call KDevelop::DUChain::self()->storeToDisk()

[Manvydas Šliamka]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #2 from Manvydas Šliamka <***@gmail.com> ---
Created attachment 105746
--> https://bugs.kde.org/attachment.cgi?id=105746&action=edit
Heaptrack results

[Manvydas Šliamka]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #3 from Manvydas Šliamka <***@gmail.com> ---
I'm really sorry for not replying sooner, but the defense of my Master's thesis
is soon and I just do not have the time to experiment with KDevelop
extensively.

First of all - tabs. I usually have 6-8 tabs open. Sometimes more, sometimes
less, but it does not seem to influence memory consumption all that much.

Next, I ran KDevelop with heaptrack in terminal, using:
heaptrack ./KDevelop.AppImage

I'm attaching the results file. Unfortunately, I'm not sure if it's complete or
not because I had to kill KDevelop. The moment I noticed the system slowing
down, I hit the close button on the KDevelop window. It closed and the memory
use stopped increasing. However, the kdevelop process remained alive, maxing
out a single CPU core. I waited for around 20 minutes, hoping for it to finish,
but it did not and I had to kill it. That's when heaptrack finished as well and
produced the heaptrack.KDevelop.AppImage.5390.gz that I'm attaching.


Admittedly, I forgot to do this:
gdb --pid $(pidof kdevelop)
...
(gdb) call KDevelop::DUChain::self()->storeToDisk()

I'll do it once I have more free time on my hands.

I've also noticed that the terminal that I used to start KDevelop was fillied
with hundreds of these messages:

kdevplatform.shell: Broken text-document:
QUrl("file:///tmp/kdevelop_Pa3204.patch")

Plenty of these messages were in the terminal as well:

kdevelop.plugins.clang: Something went wrong during 'clang_codeCompleteAt' for
file "ONE_OF_MY_CURRENTTLY_OPEN_PROJECT_FILES"
libclang: crash detected in code completion

[Milian Wolff]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #4 from Milian Wolff <***@milianw.de> ---
The way AppImage works makes it impossible to track kdevelop with heaptrack
currently. So you'll either have to runtime attach heaptrack later on
(heaptrack --pid $(pidof kdevelop), assuming an AppImage client application can
be found that way).

And don't worry with the delay, we too have lots of stuff to do. So take your
time and concentrate on your thesis. Good luck!

[ddz]

https://bugs.kde.org/show_bug.cgi?id=379551

ddz <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@gmail.com

--- Comment #5 from ddz <***@gmail.com> ---
Created attachment 110818
--> https://bugs.kde.org/attachment.cgi?id=110818&action=edit
htop

htop screenshot for appimage high memory use

[ddz]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #6 from ddz <***@gmail.com> ---
Created attachment 110819
--> https://bugs.kde.org/attachment.cgi?id=110819&action=edit
heaptrack(2018-02-20)

heaptrack.kdevelop.17748.gz

[ddz]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #7 from ddz <***@gmail.com> ---
Hi, I have the same issue.

My system info is

No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 9.3 (stretch)
Release: 9.3
Codename: stretch

Linux ddzt 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64
GNU/Linux

I am using KDevelop-5.2.1-x86_64.AppImage

I did this "heaptrack --pid $(pidof kdevelop)", I killed the kdevelop manually
after a few mins.

https://bugs.kde.org/attachment.cgi?id=110818
https://bugs.kde.org/attachment.cgi?id=110819

[Milian Wolff]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #8 from Milian Wolff <***@milianw.de> ---
that heaptrack trace is pretty meaningless - it only shows stuff in deflate and
not a lot of memory consumption in total. Are you sure the pid it attached to
is the one you saw in htop? Also, just to ensure, please build heaptrack from
sources, there was a bug in the attach mode which may explain the lack of data:
https://phabricator.kde.org/R45:779dceb8cbe77a2d653d5244126a80e0d5d5346f

[ddz]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #9 from ddz <***@gmail.com> ---
Hi, I am sorry, the htop screenshot and the heaptrack not create at the same
time.

I am keep running heaptrack with kdevelop today, but seem not happened today.
(Maybe I cleaned the cache, when I open kdevelop there have a dialog)
The final value are VIRT 9122M, RES 2857M, SHR 444M.

(Before, the VIRT can reach to 15G+)

I will keep tracking. If I have a good track file, I will put it here.

Thank you very much!

[ddz]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #10 from ddz <***@gmail.com> ---
Also, I build heaptrack from this commit
https://github.com/KDE/heaptrack/commit/6e31841708b351b9f53e59d384e94df014d927ed

[ddz]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #11 from ddz <***@gmail.com> ---
It's happened today, I just open the kdevelop and left it there.
Several hours later, I come back my desk, try to do something.
I feel my computer have a little freeze (cursor will stuck a few second)
Then I check the htop, I saw VIRT 14G+, RES 10G+ (around 1min, RES reduce to
8G+, then I feel the computer is come back)

I click the kdevelop's close button, but still running. So I pressed "ctrl+c"

I will put the screenshot and heaptrack here. Hope it's can help you.

THX

[ddz]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #12 from ddz <***@gmail.com> ---
Created attachment 110870
--> https://bugs.kde.org/attachment.cgi?id=110870&action=edit
heaptrack(2018-02-21)

[ddz]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #13 from ddz <***@gmail.com> ---
Created attachment 110871
--> https://bugs.kde.org/attachment.cgi?id=110871&action=edit
htop(2018-02-21)

[ddz]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #14 from ddz <***@gmail.com> ---
Created attachment 110872
--> https://bugs.kde.org/attachment.cgi?id=110872&action=edit
after_click_close_btn

[ddz]

https://bugs.kde.org/show_bug.cgi?id=379551

ddz <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Version|5.1.0 |5.2.1

--- Comment #15 from ddz <***@gmail.com> ---
Sorry, The size of heaptrack.kdevelop.22591.gz file is 7.8 MB, So I compress it
to 7z file.

[Milian Wolff]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #16 from Milian Wolff <***@milianw.de> ---
That file is equally uninteresting and also only shows deflate stuff. Heaptrack
only saw a memory peak of 2.7 *mega*byte, so I guess attaching heaptrack to the
AppImage really doesn't work at all...

One more thing you could try:

- unpack the appimage with --appimage-extract
- go into the squashfs-root folder, edit the AppRun script to run kdevelop
through heaptrack
- launch the AppRun script

maybe we get more information that way?

[ddz]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #17 from ddz <***@gmail.com> ---
I will try it. Thanks.

[Milian Wolff]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #18 from Milian Wolff <***@milianw.de> ---
Hey there,

I was just notified about this:

https://codereview.qt-project.org/#/c/218906/

So maybe that's the issue you are facing. Do you see messages about crashes on
the command line output?

[ddz]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #19 from ddz <***@gmail.com> ---
Hi,

I don't see any "crash" keywords in terminal.

Also, I found if I did the "clear cache" when launch kdevelop.

Even run 10+ hours, the max memory usage VIRT 8922M, RES 3493M, SHR 32440

At least, my computer don't freeze.


I can put the log file here.

[ddz]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #20 from ddz <***@gmail.com> ---
Hi,

I don't see any "crash" keywords in terminal.

Also, I found if I did the "clear cache" when launch kdevelop.

Even run 10+ hours, the max memory usage VIRT 8922M, RES 3493M, SHR 32440

At least, my computer don't freeze.


I can put the log file here.

[ddz]

https://bugs.kde.org/show_bug.cgi?id=379551

ddz <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #110818|0 |1
is obsolete| |
Attachment #110819|0 |1
is obsolete| |
Attachment #110870|0 |1
is obsolete| |
Attachment #110871|0 |1
is obsolete| |
Attachment #110872|0 |1
is obsolete| |

--- Comment #21 from ddz <***@gmail.com> ---
Created attachment 111039
--> https://bugs.kde.org/attachment.cgi?id=111039&action=edit
kdevelop log

[Martin Fuhrer]

https://bugs.kde.org/show_bug.cgi?id=379551

Martin Fuhrer <***@alumni.ucalgary.ca> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@alumni.ucalgary.ca

[Martin Fuhrer]

https://bugs.kde.org/show_bug.cgi?id=379551

Martin Fuhrer <***@alumni.ucalgary.ca> changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #105746|0 |1
is obsolete| |

--- Comment #22 from Martin Fuhrer <***@alumni.ucalgary.ca> ---
Created attachment 112098
--> https://bugs.kde.org/attachment.cgi?id=112098&action=edit
Heaptrack file (PID 12200)

[Martin Fuhrer]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #23 from Martin Fuhrer <***@alumni.ucalgary.ca> ---
Created attachment 112099
--> https://bugs.kde.org/attachment.cgi?id=112099&action=edit
KDevelop log (PID 12200)

First and last 1000 lines (full log is 45 MB)

[Martin Fuhrer]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #24 from Martin Fuhrer <***@alumni.ucalgary.ca> ---
I have been testing the KDevelop 5.2.1 AppImage on Debian 7.8 (wheezy) for the
last few weeks (our office is still using KDevelop 4.7.4, which runs great,
albeit a bit slow with large projects, and it would be nice to move to the
latest). For the most part, KDevelop 5.2.1 is runs along with no problems
(sometimes for one to two days at a time), but occasionally it starts eating
away at memory rapidly, to the point where my desktop starts freezing up and I
have to type Ctrl-Alt-F1 to switch to console mode and kill the kdevelop
process. My workstation has 32 GB RAM, and after I kill the process, usually 20
GB or more becomes available again.

The last few days I have been launching KDevelop with the latest heaptrack
(built from master). Rather than launching the AppRun script, I simply set the
environment variables from my shell (csh), and launch KDevelop with:

heaptrack kdevelop >& kdevelop_log.txt

This afternoon I encountered the memory leak once again. Managed to select
Session > Quit from KDevelop, and though the window disappeared, the kdevelop
process (PID 12200) kept running and consuming additional memory. Eventually I
sent two SIGTERM signals to kill it. I have attached the heaptrack file and the
first and last 1000 lines of the log file. Hopefully there is some helpful
information here.

[Milian Wolff]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #25 from Milian Wolff <***@milianw.de> ---
Martin, how did you run heaptrack? The track file only sees 27.7MB and it only
recorded for 9.7 seconds. Did it really only run for such a short time?

[Martin Fuhrer]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #26 from Martin Fuhrer <***@alumni.ucalgary.ca> ---
Strange, heaptrack was running the kdevelop session for around eight hours (and
kdevelop was running smoothly for most of this time, using a stable amount of
memory when I checked a couple times during the first few hours). A few
additional details I forgot to mention yesterday:
* I unpacked the AppImage and ran kdevelop from the usr/bin/ directory within.
* The default Debian 7 gcc (v4.7.2) is too old to build heaptrack, so I used a
custom install of the gcc v4.8.5 compiler and runtime for the heaptrack session
(kdevelop exhibits the memory leak whether it uses the gcc v4.8.5 runtime or
the default v4.7.2 runtime).
* When the kdevelop process refused to quit (and continued eating up memory), I
sent SIGTERM to the kdevelop process (not the heaptrack process).

Here are all the commands I ran (from csh):

set DIR="/scratch/home/mfuhrer/sw/db7/src/kdevelop-5.2.1"
setenv LD_PRELOAD $DIR/exec_wrapper.so
setenv APPIMAGE_ORIGINAL_QML2_IMPORT_PATH ""
setenv APPIMAGE_ORIGINAL_LD_LIBRARY_PATH $LD_LIBRARY_PATH
setenv APPIMAGE_ORIGINAL_QT_PLUGIN_PATH $QT_PLUGIN_PATH
setenv APPIMAGE_ORIGINAL_XDG_DATA_DIRS $XDG_DATA_DIRS
setenv APPIMAGE_ORIGINAL_PATH $PATH
setenv APPIMAGE_ORIGINAL_PYTHONHOME ""
setenv QML2_IMPORT_PATH $DIR/usr/lib/qml
setenv LD_LIBRARY_PATH $DIR/usr/lib/:${LD_LIBRARY_PATH}
setenv QT_PLUGIN_PATH $DIR/usr/lib/qt5/plugins/
setenv XDG_DATA_DIRS $DIR/usr/share/:${XDG_DATA_DIRS}
setenv PATH $DIR/usr/bin:${PATH}
setenv KDE_FORK_SLAVES 1
setenv PYTHONHOME $DIR/usr/
setenv APPIMAGE_STARTUP_QML2_IMPORT_PATH $QML2_IMPORT_PATH
setenv APPIMAGE_STARTUP_LD_LIBRARY_PATH $LD_LIBRARY_PATH
setenv APPIMAGE_STARTUP_QT_PLUGIN_PATH $QT_PLUGIN_PATH
setenv APPIMAGE_STARTUP_XDG_DATA_DIRS $XDG_DATA_DIRS
setenv APPIMAGE_STARTUP_PATH $PATH
setenv APPIMAGE_STARTUP_PYTHONHOME $PYTHONHOME
setenv KDEV_DISABLE_PLUGINS KDevWelcomePage
setenv LD_LIBRARY_PATH
/vega/xp/ThirdParty/gcc/db7/xprel/lib64:${LD_LIBRARY_PATH}
cd $DIR/usr/bin
heaptrack kdevelop >& kdevelop_log.txt

The last line was executing for about eight hours (until I killed kdevelop).
Could the track file be truncated because I killed the kdevelop process? Is
there some way to monitor the track file during the heaptrack session?
Unfortunately I don't have KDE 5, so can't build the heaptrack GUI to verify
the track file.

[Milian Wolff]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #27 from Milian Wolff <***@milianw.de> ---
You can try `heaptrack_print heaptrack.*.gz |& less` - it's CLI only and
doesn't depend on any KF5 libs.

The commands look harmless, I wonder what the issue is here...

[Milian Wolff]

https://bugs.kde.org/show_bug.cgi?id=379551

Milian Wolff <***@milianw.de> changed:

What |Removed |Added
----------------------------------------------------------------------------
Resolution|WAITINGFORINFO |---
Status|NEEDSINFO |UNCONFIRMED

--- Comment #28 from Milian Wolff <***@milianw.de> ---
Just tried with the steps you provided, and I can reproduce the heaptrack bug
which is now covered there: https://bugs.kde.org/show_bug.cgi?id=393387

Let's continue the heaptrack investigation there and then once that is fixed we
can get back to the KDevelop issue here.

[Milian Wolff]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #29 from Milian Wolff <***@milianw.de> ---
ok Martin, heaptrack should now work properly! please try again. and thanks a
lot for the report, this was quite a nasty issue!

[Martin Fuhrer]

https://bugs.kde.org/show_bug.cgi?id=379551

Martin Fuhrer <***@alumni.ucalgary.ca> changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #111039|0 |1
is obsolete| |
Attachment #112098|0 |1
is obsolete| |
Attachment #112099|0 |1
is obsolete| |

--- Comment #30 from Martin Fuhrer <***@alumni.ucalgary.ca> ---
Created attachment 112317
--> https://bugs.kde.org/attachment.cgi?id=112317&action=edit
KDevelop log (PID 16665)

KDevelop messages printed to the terminal.

[Martin Fuhrer]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #31 from Martin Fuhrer <***@alumni.ucalgary.ca> ---
Created attachment 112318
--> https://bugs.kde.org/attachment.cgi?id=112318&action=edit
KDevelop backtrace (PID 16665)

Backtrace generated when I tried to quit KDevelop during memory leak.

[Martin Fuhrer]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #32 from Martin Fuhrer <***@alumni.ucalgary.ca> ---
Hi Milian, thanks for the heaptrack fix. It appears to be working correctly
now. I encountered the memory leak again this morning (workstation started
freezing up), and tried to quit KDevelop. I waited several minutes until
KDevelop crashed and attached the log and backtrace. The heaptrack file is very
large so I've posted it for download from a third party site:
https://mega.nz/#!sdMXxZBT!Pxv1UGEqS2ef73Q39IgtELcneWzwXHpgszxYeyBAmaw

Reviewing the log it appears that there is some issue while libclang is parsing
a source file. I am curious if this is what triggers the leak. I will try some
more tests on my end...

[Milian Wolff]

https://bugs.kde.org/show_bug.cgi?id=379551

--- Comment #33 from Milian Wolff <***@milianw.de> ---
It could potentially be the same reason as the one explained here:

https://codereview.qt-project.org/#/c/218906/

i.e. for some reason clang crashes, then we essentially leak memory...

[Martin Fuhrer]

https://bugs.kde.org/show_bug.cgi?id=379551

Martin Fuhrer <***@alumni.ucalgary.ca> changed:

What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |FIXED
Status|REPORTED |RESOLVED

--- Comment #34 from Martin Fuhrer <***@alumni.ucalgary.ca> ---
The clang issue in Milian's previous comment appears to have been resolved. I
noticed that the KDevelop 5.3.0 AppImage is distributed with libclang.so.6
(KDevelop 5.2.x was using libclang.so.5).

Good news is, I have been trying out KDevelop 5.3.0 for a week and haven't
encountered the memory leak anymore.