Memory usage is huge MusicBrainz Picard 3.0.0rc3

[seranmm]

Hello,

I try MusicBrainz Picard 3.0.0rc3 on windows and linux it end the same.

When adding 6800+ known files with proper musicbrainz_albumid (not unknown files) it take a while but it works. But when i now select all the releases with ctrl+a and want to save them the memory usage is huge with every file it expand to infinite.

my settings are:

• Write tags to file
• clear existing tags
• remove id3 tags from flac
• cover art just one file named cover
• remove cover from tags
• convert to jpeg

and even if i save the releases 20 or 50 by hand the memory usage expand unusale and is very high.

someone else notice something like that?

[Zas]

Can you provide actual numbers?
How do you measure memory usage exactly?

[outsidecontext]

Is the memory usage different for Picard 2? We did several optimizations, and actually memory usage should be lower on Picard 3 overall.

[seranmm]

sorry for the late answer but every post from me must be Awaiting Approval.

here are two videos one with picard 2 where everythings works and without crash and the last one with picard 3 with memory problem. Sorry i cant capture the crash proper because the system is not repson for couple of minutes but the memory bar is full in top and the journal appears the OOM error.dont know if that even helps. Watch picard2 | Streamable Watch picard3 | Streamable

[seranmm]

in terminal i just run picard from konsole when killed output is is Killed picard when try to save many release at once (select all ctrl+a) and then click save. the journal show oom reason:

Sep 20 07:56:13 archlinux kernel: QSGRenderThread invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COM
P), order=0, oom_score_adj=200
Sep 20 07:56:13 archlinux systemd[516]: app-org.kde.konsole-905.scope: The kernel OOM killer killed some processes in this unit.
Sep 20 07:56:13 archlinux systemd[1]: user@1000.service: The kernel OOM killer killed some processes in this unit.
Sep 20 07:56:13 archlinux kernel: oom_kill_process.cold+0x8/0x91
Sep 20 07:56:13 archlinux kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Sep 20 07:56:13 archlinux kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.
service/app.slice/app-org.kde.konsole-905.scope/tab(3211).scope,task=picard,pid=3225,uid=1000
Sep 20 07:56:13 archlinux kernel: Out of memory: Killed process 3225 (picard) total-vm:5066076kB, anon-rss:3384336kB, file-rss:20kB, shmem-rss:329
76kB, UID:1000 pgtables:7436kB oom_score_adj:200

im using AUR (en) - picard3 which downloads/use http://data.musicbrainz.org/pub/musicbrainz/picard/picard-3.0.0rc3.tar.gz

[Zas]

I think I found the issue, that’s not directly a memory leak, but a major performance issue causing objects to accumulate, that could explain what you observe, especially with a lot of files.
If you can test from source, just to confirm, see PICARD-3453: Speed up tag tooltips: reuse Markdown instance and cache rendered tooltips by zas · Pull Request #3462 · metabrainz/picard · GitHub

The issue is there since a1, but degraded even more in a3.

[seranmm]

im using https://github.com/metabrainz/picard/pull/3462/commits/ee56d67dce4968b3cd3fa01696f23cc65b4aed26.patch ontop of http://data.musicbrainz.org/pub/musicbrainz/picard/picard-3.0.0rc3.tar.gz but the memory usage still expand to huge that it crash when saving many files. Im using the patch in prepare() in PKGBUILD

prepare() {
patch -d "$_pkgname-$pkgver" -Np1 -i "${srcdir}/ee56d67dce4968b3cd3fa01696f23cc65b4aed26.patch.patch"
}

and build log seems okay:

==> Starting prepare()...
patching file picard/tags/docs.py
patching file picard/tags/tagvar.py
Hunk #1 succeeded at 37 (offset -1 lines).
Hunk #2 succeeded at 48 (offset -1 lines).
patching file test/tags/test_docs.py

I know its archlinux topic but cant build from git right know AUR (en) - picard-git seems unsupported and don’t build.

i just better stay at version2 picard i guess. anyway want to say that seems there is still a problem.

[Zas]

Can you provide a full debug log?
Do you have any plugin enabled?
What is the memory use when the program was just started (nothing loaded)?

[rdswift]

@seranmm, if you’re using the “Additional Artists Details” plugin, that’s almost certainly where your memory issues are coming from. It wasn’t designed with processing large numbers of files in a single session in mind, and the session cache is growing out of control. Try disabling the plugin, restart Picard, and see if that eliminates the problem.

This has caused me to re-think the design of the plugin, so I will be working on an update that is much more resilient to large batch / session processing.

[seranmm]

so one good news is im now using picard from git so latest version. commit is 618e796 so its rc4.

the bad this still happens. and i don’t use any plugins. and if do the same in picard2 no ram issue.

and the debug log is huge because many files but i dont notice anything unormal just

D: 13:28:36,551 formats/vorbis._save:268: Saving file
D: 13:28:36,778 file.update:986: Updating file <FLACFile

picard when started use not much ram then i load my files little bit more ram but when i start saving my files the more files saved the more ram is used until my ram is full and OOM kill picard. is seems that every 5-10files take 100MB ram and this never ends. here are screenshots.

picard just started

all my files loaded into picard

start saving all my files

and more and more ram usage with every file saved and this never ends

[outsidecontext]

@seranmm We are investigating this right now, and we are finding various things. One main issue is related to updating the cover image view on the bottom right when multiple files are selected. A workaround is to disable the cover art display via the menu View → Cover Art.

Can you check whether disabling the cover art view helps with your specific issue?

[seranmm]

wow this is it.

i uncheck “View > Cover Art” and now the saving also very fast and the ram usage is very minimal like picard2. and all time i just don’t enable cover download in first place its seems just the view option even when no cover download is involved.

[outsidecontext]

Yes, it is just the view option The memory usage is not due to the cover art or such, but temporary objects being created during UI updates and hold in memory as long as the files stay loaded. Working on a fix.

[outsidecontext]

As a side note, I can reproduce the general behaviour of increasing memory use also with Picard 2. It is not quite as bad, though.

A separate workaround is to immediately deselect the files, e.g. click on a single item. This will also avoid the memory issue, but also massively increase the save performance. Beside the memory issue described above, the other core issue is that the metadata + cover art view at the bottom gets updated after each save, and for many selected files this is expensive. @Zas is working in improvements for this.

Anyway, thanks for the report and the help debugging this. Much appreciated

[seranmm]

thanks to everyone who try to help and believe.

i am very happy that the time i take into it worth it and the issue is reproducible.

thanks to every dev who make picard3 a good program and i am confident that it get fixed.

[Zas]

@seranmm we merged multiple improvements already, since you were running from source you can pull latest master branch and give it a try, please report back.

[seranmm]

okay i test from Free context menus after use to release captured objects · metabrainz/picard@9de870b · GitHub and all work fine. i even enable all the other options under view. and no more memory issues or performance issues when saving. thanks !

[seranmm]

but now i notice another issue with same symptoms (huge ram usage and then picard killed oom).

when you select many releases and drag&drop releases back to “unclustered files”.

[Zas]

@seranmm

when you select many releases and drag&drop releases back to “unclustered files”.

What do you mean by “back”? Explain more clearly please.

[seranmm]

here a picture hope that clear the situation.

when you drag and drop with mouse the all selected releases (ctrl+a) back to unclustered files the issues appears same as before for every file 10-100MB ram so on until every memory is used and crash.

the use case is not normal (who would remove fine releases back) but the ram issue is real and should be fixed i guess.