# "recording" json dumps: incomplete?
**URL:** https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708
**Category:** Development
**Tags:** database, json, datadump, musicbrainz
**Created:** [August 15, 2017, 1:24pm UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708 "2017-08-15T13:24:48Z")
**Posts on this page:** 19
**Page:** 1
### Author:  [@ruben\_sav](https://community.metabrainz.org/u/ruben_sav)
#### Post date: [August 15, 2017, 1:24pm UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/1 "2017-08-15T13:24:48Z")
I’m using the json dumps of the Musicbrainz data (from [Index of /pub/musicbrainz/data/json-dumps/](http://ftp.musicbrainz.org/pub/musicbrainz/data/json-dumps)), and noticed something weird. It holds only about 60k-70k recordings, while the Musicbrainz database statistics mention there are over 17 million recordings ([Database statistics - Overview - MusicBrainz](https://musicbrainz.org/statistics)). How come they are not all in the dump? And is it possible to get a complete dump of this? The numbers for other json dumps I’ve looked at (releases, release groups, artists, labels, areas) do add up compared to the database statistics.
The “recording” dump is very useful for me to couple different music related datasets for research, since it contains ISRC numbers where available (and another dataset I use has them as well).
---
### Author:  [@Freso](https://community.metabrainz.org/u/Freso)
#### Post date: [August 17, 2017, 10:41am UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/2 "2017-08-17T10:41:36Z")
Hi! This is a bug on our end. I’ve filed [MBS-9433: JSON dump doesn’t include all Recordings](https://tickets.metabrainz.org/browse/MBS-9433) and hopefully @bitmap will be able to fix it soon. Sorry!
---
### Author:  [@Bitmap](https://community.metabrainz.org/u/Bitmap)
#### Post date: [August 17, 2017, 4:28pm UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/3 "2017-08-17T16:28:45Z")
Hi there, it actually intentionally only includes standalone recordings (I left a comment on the ticket). You’ll have to get recordings from the release dump.
---
### Author:  [@ruben\_sav](https://community.metabrainz.org/u/ruben_sav)
#### Post date: [August 21, 2017, 9:07am UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/4 "2017-08-21T09:07:32Z")
Hi! Thanks for the response. I understand why the recording dump isn’t complete now, but my problem still stands. The release dump doesn’t include the ISRC code. The recording dump is the only one to include them as far as I have been able to see, so I still don’t have a way of getting the ISRC through the dumps.
Is there any way the recording dump could be completed or the ISRC codes added to the release dump?
---
### Author:  [@Bitmap](https://community.metabrainz.org/u/Bitmap)
#### Post date: [August 22, 2017, 4:35am UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/5 "2017-08-22T04:35:42Z")
You’re absolutely right, sorry for claiming that they included ISRCs without verifying it. That was a bug which I’ve now fixed. I’ve triggered a new full dump, so I’ll link that here when it’s ready.
---
### Author:  [@ruben\_sav](https://community.metabrainz.org/u/ruben_sav)
#### Post date: [August 22, 2017, 9:55am UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/6 "2017-08-22T09:55:36Z")
Great, thanks a lot! So it’ll be in the release dump then when it’s ready?
---
### Author:  [@Bitmap](https://community.metabrainz.org/u/Bitmap)
#### Post date: [August 23, 2017, 10:53pm UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/7 "2017-08-23T22:53:25Z")
Yeah, it should. Though I had to restart the dump today since I noticed another bug (sorry).
---
### Author:  [@Bitmap](https://community.metabrainz.org/u/Bitmap)
#### Post date: [August 24, 2017, 6:03pm UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/8 "2017-08-24T18:03:15Z")
Done, the 20170823-225118 dump should have them now: [http://ftp.musicbrainz.org/pub/musicbrainz/data/json-dumps/20170823-225118/](http://ftp.musicbrainz.org/pub/musicbrainz/data/json-dumps/20170823-225118/)
---
### Author:  [@ruben\_sav](https://community.metabrainz.org/u/ruben_sav)
#### Post date: [August 25, 2017, 12:03pm UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/9 "2017-08-25T12:03:51Z")
Hi, thanks! The ISRC is available indeed. I now notice the latest release dump is a lot smaller compared to the previous one I have from 9th of August (1.31gb vs 2.7gb) and it holds less releases as well. Is this another problem?
I also saw that there is a dump from with same date as the last one. That one has a release dump of 2.71gb as I would expect. Would it cause me any problems if I use that one instead?
---
### Author:  [@Bitmap](https://community.metabrainz.org/u/Bitmap)
#### Post date: [August 25, 2017, 1:48pm UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/10 "2017-08-25T13:48:17Z")
There is definitely something strange with the file sizes that I’m investigating (it takes a while to decompress them…), but at a glance I see the opposite of what you’re saying: the latest release dump from 20170823-225118 is much larger (9.3G) than the one from 20170823-001002 (2.6G), not smaller. Can you point to exactly which file is 1.3G?
If the other dump from the same date you’re talking about is 20170823-001002, then no, that one can’t be used because it was from before I applied the fix to dump ISRCs.
---
### Author:  [@ruben\_sav](https://community.metabrainz.org/u/ruben_sav)
#### Post date: [August 25, 2017, 2:04pm UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/11 "2017-08-25T14:04:51Z")
Oh right, I was only looking at the one file with the “release” dump. The file I’m talking about is the release.tar.xz one
---
### Author:  [@Bitmap](https://community.metabrainz.org/u/Bitmap)
#### Post date: [August 26, 2017, 1:08am UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/12 "2017-08-26T01:08:15Z")
Sorry, I still don’t understand where you’re getting/seeing 1.31 GB. The latest release.tar.xz is 9.3 GB and contains more releases as expected (1848011 vs. 1847460 from the previous one).
---
### Author:  [@ruben\_sav](https://community.metabrainz.org/u/ruben_sav)
#### Post date: [September 11, 2017, 12:18pm UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/13 "2017-09-11T12:18:30Z")
Hi, sorry for my late reply. I left on vacation (thinking something just went wrong with my download after reading your last comment) without computer, so I wasn’t able to download again and check if everything worked before I got back.
Anyway, I just got back and downloaded the latest dump and still have a similar problem. In the latest json dump, the release.tar.xz seems to be 1.46gb. It seems to hold only about 200000 releases. I am really confused about what could be the difference between what you are seeing and what I am seeing.
For clarity: the file I am talking about is in the folder [http://ftp.musicbrainz.org/pub/musicbrainz/data/json-dumps/20170909-001001](http://ftp.musicbrainz.org/pub/musicbrainz/data/json-dumps/20170909-001001) (logging in as a guest when prompted). If I check the previous folder (20170906-001001), the file I’m seeing is 1.9gb.
 
---
### Author:  [@Bitmap](https://community.metabrainz.org/u/Bitmap)
#### Post date: [September 11, 2017, 10:06pm UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/14 "2017-09-11T22:06:04Z")
Yep, that’s very strange. I also have a Mac (10.12.6), and it shows 1.9GB in Finder’s FTP window. But I can confirm it’s 9.4GB on disk, which is also what the OSUOSL web interface reports, so I assume it’s synced correctly. It could be hitting an ancient 2GB limit somewhere. Can you download it via curl instead? curl also reports the total as 9583M for me.
---
### Author:  [@Bitmap](https://community.metabrainz.org/u/Bitmap)
#### Post date: [September 12, 2017, 4:47am UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/15 "2017-09-12T04:47:45Z")
Regarding the size increase (from 2.6G to 9.3G): I changed the tool used to compress the dumps, and created a new dump (20170911-234731), and the release one is back to 2.5G, so it seems to have been an issue in pixz.
As I’m writing this the 20170911-234731 dump isn’t synced to FTP yet, but it should be soon.
---
### Author:  [@ruben\_sav](https://community.metabrainz.org/u/ruben_sav)
#### Post date: [September 12, 2017, 2:39pm UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/16 "2017-09-12T14:39:29Z")
Okay, I see, I only recently updated my Mac to 10.12 so that’s probably the problem indeed. Let me try using curl!
---
### Author:  [@ruben\_sav](https://community.metabrainz.org/u/ruben_sav)
#### Post date: [September 13, 2017, 7:32am UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/17 "2017-09-13T07:32:13Z")
Since curl is new to me, this might be a stupid question: is it possible to use curl as a guest user? I keep running into errors.
I tried issuing the following command:
`curl ftp://ftp.musicbrainz.org/pub/musicbrainz/data/json-dumps/20170906-001001/release.tar.xz -o release.tar.xz`
Resulting in the following error:
“curl: (9) Server denied you to change to the given directory”
I’ve tried setting the user to guest, but then get the following:
“curl: (67) Access denied: 530”
---
### Author:  [@dns\_server](https://community.metabrainz.org/u/dns_server)
#### Post date: [September 13, 2017, 7:47am UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/18 "2017-09-13T07:47:49Z")
That directory no longer exists but i can see a dump from today.
curl [ftp://ftp.musicbrainz.org/pub/musicbrainz/data/json-dumps/20170913-001001/release.tar.xz](ftp://ftp.musicbrainz.org/pub/musicbrainz/data/json-dumps/20170913-001001/release.tar.xz) -o release.tar.xz
---
### Author:  [@ruben\_sav](https://community.metabrainz.org/u/ruben_sav)
#### Post date: [September 13, 2017, 7:51am UTC](https://community.metabrainz.org/t/recording-json-dumps-incomplete/322708/19 "2017-09-13T07:51:21Z")
Right, that works, thanks! I checked if the directory existed when I started looking into this, and while I solved my stupid previous errors, it must’ve been replaced 🙂