On the cause of hangs/crashes


#1

There have been a number of posts in the past about Picard hanging/crashing when attempting to process a large number of files. Suggestions to address this have included breaking up the music files into smaller batches. I have been doing a lot of testing recently of my Classical Extras plugin, which has involved a fair number of writes to the log file. My observation is that Picard will hang if there are an excessive number of writes to the log file (in the order of hundreds of thousands). I have run the same code with and without the logging against the same file sets and it will hang if the logging is turned on and not hang if it is off. This behaviour has occurred identically with various versions of the code, log writes and file sets.
So maybe the cause of hanging with large numbers of files is nothing to do with the main Picard code, but merely a function of the number of log writes. Note that the total size of the log file seems to be irrelevant - it is the number of writes in the current session that seems to be critical.
Has anybody else noticed this? Perhaps someone could run an independent check of my hypothesis? Or better still, fix it?


Picard is extremely slow on larger amout of songs
#2

I experience this issue more often than not. But it happens to me at about 2000 songs. It was the reason I enabled logging. My theory - I am just a novice - was a process is hanging
I see four msvcr90.dll!endthreadex+0x6f processes just spinning. So I increased the priority of these to “above normal” and set picard.exe to “highest” (NOT real-time) and in a few minutes, the app was responsive again. This is the first time i have tried this and I’m sure results will vary - if duplicated at all.
I didn’t know the app was written using python until now so there may be other places to look.
I’ll post any additional results - because it’s bugging me even more now.

-john


#3

@wehavecookies4u, can I ask how long it took to tag those 2000 songs?


#4

Tried that when Picard seemed totally unresponsive and after a few minutes it recovered!


#5

Great. I have times where that hasn’t worked. (or i haven’t given enough time)
Large multi-track mp3’s (oversight) will also cause an issue.
A stop button would be ideal with a message of what track was interrupted.
More baffling is the inconsistent behaviour with the same tracks on two different machines.
i started to set up for a trace dump but a preview update caused some time without the machine. The other machine (both win 10) seems to be banging thru with a few hangs. I started to break up the albums into smaller sets and it runs thru. It starts getting SLOW. Clearing .\AppData\Local\MusicBrainz\Picard\cache\picard and temp seems to fix that.
Regards,
John


#6

How do you turn the logging off?


#7

Not sure you can turn it off, but you can make sure that the “debug mode” box is not checked (go to Help->View Error/Debug log).


#8

3 posts were split to a new topic: Random crash reports


#9

There were a lot of reports of slowdowns with huge collections.
There are many reasons for that, some just make sense (like many queries + rate limiting = slow or tagging huge files that need to be fully rewritten on slow network shares).
We also know UI can hang, mainly due to Python / Qt / threading issues. This will be addressed in future releases, but it implies a lot of changes in the code.

That said, most slowdown reports come from windows users, and almost none from linux / macosx users, so i suspect something related to the win version only.

Picard is, by itself, logging to stderr (on linux), but it seems it logs to a file in the packaged windows version.

Of course, debug mode is very verbose (that’s its purpose), people aren’t expected to use it all the time.
But it causes no slowdown on linux systems. The write rate is quite low by today’s standards.

It should be noted that all read or write operations from/to audio files are made through mutagen library.
It should be also noted, mutagen has to rewrite files in full to insert metadata in worst cases (it depends on size of metadata vs space reserved in the file).
Also some users are embedding full size images from CAA, that’s a damn bad idea (some images are > 50MB in size).
Of course, if one is reading / writing from network shares, it is even slower (especially on SMB).

So, digging this problem is a good idea, but please indicate exact environnement, software versions, disks and/or share, network speed, types of files, etc…

Logging network (http queries to web service), disk ops, etc, can also help.


#10

FWIW, just as regards the logging issue: I was using debug mode quite extensively when developing the “Classical Extras” plugin and it was definitely causing problems (W10). I’ve now given the plugin its own custom logging feature, which is even more extensive but does not cause any hanging issues. It just writes to a file with the default buffering, which I think is a different approach to Picard. (Also separate log files are written for each release, to aid debugging, which helps to keep the file sizes down too).


#11

Which version of Picard are you writing about ?
FYI, upcoming Picard 2 logging system changed.


#12

Sorry, 1.4.2
Will look at moving onto 2 in a month or so.