MusicBrainz not fully indexed by Google

I’ve a small concern or two.
Can the servers handle a ? 25% ? increase in traffic over the next 3 months?
Have “Events” also been “not indexed”? Cause if Events soon started to take off and Musicbrainz became the global go-to live music event guide - I think the numbers would be large.

Musicbrainz global gig guide - go live!

2 Likes

If events did take off, we’d add more capacity to our servers, for sure. I’d love to have this problem. :slight_smile:

10 Likes

Myron Rosmarin on Quora wrote: "So, does more content on the web make it easier or harder to choose the 10 best? I would suggest it makes it harder. This is one of the main motivators Google has for limiting the size of their searchable index to just the absolute best content."
This “limiting … to the absolute best content”, if the explanation for the non-indexing of many MusicBrainz pages, would also be a good explanation for why I find Discogs pages that are not indexed either, and notice a general deterioration in finding useful niche search results on Google.

1 Like

Googlebot is currently crawling ~6.5 milions of our pages per day. We can expect better search results soon.

6 Likes

And their usage of “most relevant to you” results don’t make it any better. Local results. Cookies. I want to search the internet!
I sometimes use a VPN to change my location to get different search results.

Apparently, I am very popular on Russian websites. I have more press over there than I do in America. I should consider a tour.

4 Likes

Doesn’t appear to be much improved yet. Anyone else seeing improvements?

(I see my MBS ticket hasn’t been closed yet)

I have seen things that were not previously there - and things that were there but have moved up to the head of the line (page 1 instead of page 6).
That doesn’t mean that the changes are there because of the recent data mining. They could have just as easily been changed because of normal usage.

Either way, the answer is - YES.

1 Like

Would we have better luck trying to get DuckDuckGo to index us?

MB is already on DDG. I do believe they have fewer results than Google.
But DDG does give results that do not show in Google.

They definitely seem to currently have the same situation as MB on Google – the first test search I did on DDG gave no MB results.

I made a tweet asking DuckDuckGo about it:


Let’s see if they respond. :slight_smile:

5 Likes

And they did! … now we need to ask Yahoo the same, it seems :stuck_out_tongue:

4 Likes

I looked into Yahoo!, and it seems like they in turn are using Bing. So I made a Microsoft account and submitted MusicBrainz to their webmaster site thing, now just waiting for @zas to verify it. Hopefully we should be fully (or at least better) indexed by Bing/Yahoo!/DuckDuckGo soon too.

3 Likes

Looks like Morlox at least is better indexed with DuckDuckGo now:
https://duckduckgo.com/?q=morlox+site%3Amusicbrainz.org

2 Likes

But not fully… they still don’t have the events. Oh, well…progress, I suppose.

1 Like

Another way of improving the search at least on DDG might be to make an instant answers module(?) for MusicBrainz so that inquiries like “who wrote {work name}?” and “how many tracks does {release} have?” can return answers from our relationships. That is, of course, if the dev team has time to do this.

1 Like

Taking a look at that Google search you just typed:

morlox site:musicbrainz.org

Around when you googled that near September 5, did the above search not return any results? If I Google “morlox site:musicbrainz.org” now I get 3 results:

  1. Morlox - MusicBrainz
  2. Morlox - Recordings - MusicBrainz
  3. Morlox - Aliases - MusicBrainz

and if I make a search on DuckDuckGo with “morlox site:musicbrainz.org” I get 4 results:

  1. Morlox - MusicBrainz
  2. Herpes Hideaway - MusicBrainz
  3. Syphilis Sauna - MusicBrainz
  4. Patrick Urn - MusicBrainz

If I make a search on startpage.com for “morlox site:musicbrainz.org” I get 0 results



Now if I Google my artist name “zczero site:musicbrainz.org” I get 2 results(those 2 results are related to my participation in songfight.org music competitions):

  1. Release “Darling, You're Strange” by SONG FIGHT! - MusicBrainz (Release “Darling, You’re Strange” by Various Artists)
  2. Release “I Don't Like Crowds” by SONG FIGHT! - MusicBrainz (Release “I Don’t Like Crowds” by Various Artists)

Searching on DuckDuckGo.com for “zczero site:musicbrainz.org” turns up 1 result:

If I make a search on startpage.com for “zczero site:musicbrainz.org” I also get 0 results.

Though if I make a startpage.com search for “michael jackson site:musicbrainz.org” I get 2 pages of results

This may inspire me to maybe start doing events to see if they get notable enough for Google and DuckDuckGo…hmm…

I definitely don’t remember exactly what it returned, but I was most concerned that it did not return info about events that Morlox performed in.
Also, someone was kind enough to add a screenshot to the bug I created, so that will give an idea of what Google had back around that time.

I also wonder if the increase in Morlox results is due to some overall systemic improvement or just because we’ve been mentioning it here and Google’s been spidering this forum. If it’s the latter, then searching Google for artists not mentioned on the forum would still presumably produce little results.

I did a comparison using site:musicbrainz.org as part of the search criteria

Oddly enough, I got the most hits from Yahoo and Bing.

A quick update on where we stand with indexing on Google:

  • In October Google picked up indexing and we’re now at 22,016,503 pages indexed.
  • In December more pages got added and 33,994,282 URLs have been submitted for indexing, and the crawl rate has picked up again.

Once we get to the 33M value, we’ll be caught up again and roughly where we should be.

4 Likes