GSoC 2018: Add support for Reviewing & Rating more Entities

Personal information


Name: Divya Prakash Mittal
IRC nick: dpmittal
Email: divyaprakashmittal@gmail.com
GitHub: dpmittal
Personal Blog: dpmittal.me
LinkedIn: Divya Prakash Mittal
Time Zone: UTC+05:30

Project Overview


After the rating system being implemented in CritiqueBrainz, it is being planned to migrate the ratings which already existed on MusicBrainz to CB. This change will be more effective when all the entities existing on MB can be rated/reviewed on CB.

Presently, on CB, a user can review/rate only three entities i.e. Release Groups, Events and Places. By allowing them to rate/review more entities like Recordings, Works, Labels and Artists, not only it will help to migrate the ratings of MB to CB but it will also target the interest of more people and engage them to use CB.

Being related to music which is an art in itself, I feel that the current UI needs to be more vibrant which will keep the existing reviewers engaged as well as attract the interest of new users.

As my Google Summer of Code project, I aim to extend the functionality of CB by adding support of reviewing more entities i.e. Artists, Recordings, Works and Labels and migrate the ratings which existed on MB to CB as well as improve the current UI of the system.

The following implementation is in accordance with the issue CB-270 and discussion on IRC.

Deliverables

My proposed idea is divided into two major objectives:-

  1. Implementation of CB-270. - Adding support for Artists, Recordings, Works and Labels in the current review/rating system.
  2. Migrate the ratings that existed on MB to CB:- This implementation is in accordance with the issue CB-245.
  3. Improving the UI - The present UI is very much cluttered and it needs improvement as a vibrant UI will engage the users with the project. More details are given below.

Optional Ideas - Planning to make a Statistics Page for CB based on user reviews using Data Visualization and the project will be based on d3.js.

Implementation


1. Implementation of CB-270 :-

Database Queries

The database of MusicBrainz is already present in the docker-image musicbrainz_db.

Database Query functions have to be written for getting entity info from the database for artists, labels, work and recordings. This can be done using joins on the various tables like it is being currently implemented for the existing entities. Other relationships and related entities can be retrieved by separate query functions. After making queries, the retrieved data will be cached as it is being done presently for other entities.

The following tables have to be joined for respective entities for data to be retrieved:-

  • Works - For getting information like id, name, type, ISWC ; tables work, isrc and work_type can be joined and the data can be retrieved.
  • Recordings - In this case, id, name, ISRC, artists and artist ids need to be retrieved and this can be done by joining artist_credit, recordings, ISRC, artist_credit_name can be joined and then the query can be done at once.
  • Labels - For Labels, tables labels, labels_type need to be joined for data retrieval.
  • Artists - Queries for artists are already present in the codebase and only reviewing functionality has to be added to it.

In frontend/external/musicbrainz/ the following files will be added to access data from DB:-

  • label.py -
    • get_label_by_id(): To fetch data of the label such as name and other details using its mbid.
    • get_release_rels(): To fetch the releases by the label.
    • get_multiple_rels(): To fetch various artists, work, places related to the label.
  • work.py - Functions like
    • get_work_by_id(): To fetch data of the place such as name and other details using its mbid.
    • get_multiple_rels(): To fetch various artists, work, places related to the work.
  • recording.py - Functions like
    • get_recording_by_id(): To fetch data of the recording such as name, length using its mbid.
    • get_multiple_rels(): To fetch various artists, work, places related to the recording.

There are examples of database queries for Recordings and Works respectively for better understanding:-

In frontend/external/musicbrainz/tests/ folder various tests will be added to test the above functions.

Making changes to the UI for the new entities
  • Entity Pages:- There should be entity pages for artists, labels, work, and recordings; displaying the average rating of all the reviews on that particular element. For this views need to be added in critiquebrainz/critiquebrainz/frontend/views/ to render pages related to the new entity.

The entity pages which have to be rendered will be stored in /critiquebrainz/frontend/templates/. Following additional files have to be added to the folder:-

  • /labels/entity.html
  • /works/entity.html
  • /recordings/entity.html

These pages will display the information related to that particular entity and the reviews that have been made on it.

Here are examples of entity pages as they will be shown on CB:-

Artist :-

Label :-

Note:- The entities recording and work will have similar pages as shown above.

  • Review Pages for new Entities:- Have to make changes in the review editor for the user to review the new entities. Inside /frontend/templates/review/entity as well as in /frontend/templates/review/modify following new files have to be added:-
    • artist.html
    • label.html
    • work.html
    • recording.html

Modifications are required in /critiquebrainz/ws/review/views.py to facilitate the changes for new entity types. VALID ENTITY TYPES table schema needs to be updated too.

Here are examples of review editor allowing users to review new entities:-

  • Able to browse reviews by new entity types.

2. Implementation of CB-245:-

Presently on MB, five entities can be rated i.e. artist, label, recording, release_group & work. The average ratings and rating count of each entity exist in *_meta tables. For each entity, a function can be added to their Database query files which exist in frontend/external/musicbrainz/, which further can be used in avg_rating.py file while calculating the average rating for a particular entity. Here is an example gist of the function that will fetch ratings of a particular recording from the recording_meta table:-

Further, the functions in the database query files can be used in avg_rating.py in critiquebrainz/db as:-

if entity_type=="release_group":
    rating_mb, rating_count_mb = db_release.get_rating_by_id(mbid)
elif entity_type=="artist":
    rating_mb, rating_count_mb = db_artist.get_rating_by_id(mbid)
elif entity_type=="recording":
    rating_mb, rating_count_mb = db_recording.get_rating_by_id(mbid)
elif entity_type=="work":
    rating_mb, rating_count_mb = db_work.get_rating_by_id(mbid)
elif entity_type=="label":
    rating_mb, rating_count_mb = db_label.get_rating_by_id(mbid)

Afterwards, final rating and rating_count can be calculated by taking the average of ratings and summing up the rating count of both CB and MB.

The data of the editors i.e. users who have rated the entities on MB can be fetched from the *_rating_raw tables. The example function to fetch editor information like name, mbid along with their individual ratings is present in the gist above in get_rating_users_by_id function.

There will be three possibilities which have to be taken into account while calculating average ratings as shown in the diagram below:-

(i). User has rated the entity on MB only:- In this case, we would take the user’s rating on MB into account while calculating the average rating.
(ii). User has rated the entity on CB only:- In this case, we would take the user’s rating on CB into account while calculating the average rating.
(iii). User has rated the entity on both CB and MB:- In this case, it would be better to give preference to the rating that existed on CB. Hence, we would take the user’s rating on CB into account while calculating the average rating.

Note:- Here, “Rated the entity on MB” means that the user has rated the entity as well as allowed other users to see their ratings.

3. Improving the UI:- The current UI needs improvement. Here are some of my ideas for homepage and user pages respectively.

  • Homepage:- Presently, all the reviews are being shown in a cluttered manner. The current design doesn’t give a user a quick idea about the various features available on CB. For example:- In the present homepage design, a viewer has to go to different reviews to check their ratings.

In my proposed design, the user can view the ratings of the review on the homepage itself and he/she need not go on the review page. Moreover, a 3-D carousel will be a more attractive way of listing reviews.
This is my proposed design for Homepage:-

  • User Pages:- The present UI of the user page has a tabular pagination system for User information and User reviews. The redesigned page allows everything to be on a single page. The User information can be contained inside a banner and the user reviews can be paginated below with their excerpts and ratings.

Here is the proposed design for User pages:-

Note:- All the UI mock-ups depicted in the proposal are mere representations of how I envision the changes to be incorporated that suits the current design of the site. There may well be a better suggestion and final implementation at the completion of the project could be different.

Proposed Timeline


Community Bonding (April 24 – May 13):-
In this period, getting myself familiar with the environment, I will fix existing bugs of CB. I will be in constant touch with the mentor discussing and finalizing the roadmap for the project.

Phase 1 (May 14 - June 11):-
In this period, I aim to complete the database access functions for Artist and Labels entity type, and make necessary changes to the UI allowing users to review both of the entities. After completing their implementation I will start on with implementing review system for Work on CB.

Phase 2 (June 12 - July 9):-
In this period, I will complete the implementation of the Work entity as well as Recording entity on CB. I’ll also start working on migrating the ratings from MB to CB i.e. writing database query functions.

Phase 3 (July 10 - August 6):-
In this period, I will complete migrating the ratings from MB as well as make the necessary changes in UI for it and will be making changes to the UI as per decided before. I will be working on improving user experience and I will be also spending time documenting stuff and fixing bugs.

After GSoC:-
Will continue to work on CritiqueBrainz and on other MeB projects too. Planning to make a Statistics Page for CB based on user reviews using Data Visualization and the project will be based on d3.js.

Detailed Analysis of my Timeline:-

Week 1:- Complete writing database query functions for Artists and tests for them. Will also start making necessary changes to the UI for the new feature.
Week 2:- Will complete the changes to UI so that the review system for Artists is ready. Will also start writing query functions for Labels.
Week 3:- Complete writing the DB query functions for Labels as well as tests for them and will start implementing necessary changes to the UI for Labels entity review system.
Week 4:- CUSHION WEEK: Will be completing all the stuff and fixing the bugs if any, and prepare for the Phase 1 evaluations.

Phase 1 evaluations

Week 5:- Working on the feedback from the mentors and complete integrating Frontend and Backend of the Works entity type.
Week 6:- Complete the Database query functions along with their tests for Recordings and start integrating it with the frontend.
Week 7:- Fixing all the bugs if any, and start writing queries for migrating ratings from MB to CB.
Week 8:- CUSHION WEEK: Complete any pending work, (if any) and Start discussing and finalizing with mentors about the New UI by making wireframes and mockups and prepare for Phase 2 evaluations.

Phase 2 evaluations

Week 9:- Complete writing queries for migrating rating from MB. Will also start making necessary changes to the UI for the new feature.
Week 10:- Complete adding support for the migrated ratings and start working on the Finalized UI mockups/wireframes as per discussed with the mentor.
Week 11:- Finish all the work on proposed UI and fix existing bugs if any.
Week 12:- Finish Documentation and start working on Optional ideas.
Week 13:- Pencils down week. Prepare for final submission making sure that everything is OK.

About Me


I am a first-year Electronics undergrad at Thapar Institute of Engineering & Technology, Patiala. I came across the world of open-source during Google Code-in 2014. In 2015, I came across MetaBrainz due to my love for music and since then, I have been contributing to various projects of this organization. I was also selected as a finalist in both Google Code-in 2015 & 2016 for my contributions. Here is my list of PRs to MusicBrainz, BookBrainz and CritiqueBrainz. I also have a blog on which I seldom post but I plan to use it regularly through the programme blogging about my progress with the project.

Question: Tell us about the computer(s) you have available for working on your SoC project!
Answer: I have an HP laptop with Intel i3 processor with 8GB Ram and 1TB HDD running a dual boot of Windows 10 and Ubuntu 16.04 LTS.

Question: When did you first start programming?
Answer: I was introduced to QBASIC in my 6th Grade in school and to JAVA in my 9th Grade. I started on with Python mostly in 10th Grade and since then I am trying to make small projects and Web applications.

Question: What type of music do you listen to?
Answer: I mostly listen to romantic Indian tracks. Arijit Singh and Atif Aslam are my favourite singers.
I also like Ed Sheeran and The Chainmokers. Here are some of my favorite albums/songs:-

Question: What aspects of CritiqueBrainz interest you the most?
Answer: I like to watch movies, read books and mostly I love to listen to music. But most of the time I find out that a person’s review about something is different from the famous critics. Here, CB makes me interested. In CB, one can write his own reviews and browse reviews written by people who have similar taste in music and this makes me love it.

Question: Have you ever used MusicBrainz to tag your files?
Answer: No, but I will definitely try it out.

Question: Have you contributed to other Open Source projects? If so, which projects and can we see some of your code?
Answer: I have made code contributions to MetaBrainz Foundation only till now about which I have written earlier but I also did some contributions to Sahana Software Foundation by reporting bugs on their tracker and by making changes to their wiki.

Question: What sorts of programming projects have you done on your own time?
Answer: I have made a small quiz game in Java and was working on a Flask based website for my school.

I also made a website which translates a webpage when its link is entered in various Indian languages in a hackathon. This project was based on Flask, BeautifulSoup of Python. Here’s the link (there may be some glitches in it).

I am also in the team working on the college’s computing society’s Flask based website. Here’s the link (This is in its initial stages).

Question: How much time do you have available, and how would you plan to use it?
Answer: I have my summer holidays during most of my GSoC period and I plan to work full time(40-50 hrs per week) on the project.

Question: Do you plan to have a job or study during the summer in conjunction with Summer of Code?
Answer: No, if selected for GSoC.

3 Likes

This is an initial draft of my application. Open to any kind of feedback/suggestions! :slight_smile:

I have made improvements to my proposal as per @iliekcomputers tips on IRC https://chatlogs.metabrainz.org/brainzbot/metabrainz/2018-03-10/?msg=4136090&page=3 :smiley:

Hi @dpmittal, sorry for being a bit late on making reviews to your proposal. First of all, thanks for the proposal, this is a important project for CB. I’ve given your proposal a read through and it seems solid in intentions, BUT it lacks a bunch of implementation details. You should try to include as many details as possible. Right now, you’re only talking in generic terms, which makes it hard for us to estimate your understanding of what the project entails exactly.

I’ve made a few general comments but all of them share the same theme, to be honest.

This is one of the more important part of the proposal and needs a lot more detail. What data exactly will you need to find? What tables exactly will you query? What tables will you need to join? Will you cache the data somewhere? You should provide as many examples as possible and make everything as explicit as possible.

Again, a lot more detail is needed here to show that you understand this part of the codebase and the changes that you’ll need to make. You should explain what views you’ll add etc.

What changes exactly? Remember, we want to ensure that you know what problems you’re proposing to solve and how to solve them. The more details you have in your proposal, the easier it makes for us to do that.

This requires a LOT more detail. What improvements are needed? Why are they needed? How do we plan to improve the UX exactly? Is this anything other than personal preference? In general, UX changes should go through the community before being made. You should take a look at how @chhavi’s workflow for UX design work and see if you can incorporate it here somehow.

Also, what page is the llast mockup representing?

Ideally this should be done in the community bonding period. The coding period should be spent writing actual code which pushes the project ahead.

you don’t list any optional ideas.

Another thing I’m not sure about is whether adding these four entities would be enough work for a full time developer working for three months. Maybe we should consider extending the project a bit to include more entities OR somehow importing the ratings from MB into CB?

First of all Thanks for the review. I will update the proposal soon. :slight_smile:

@iliekcomputers Updated the proposal in terms of technical implementations and added a few more mockups for UI changes :slight_smile:

Hey, @dpmittal, thanks for updating your proposal! I still have some more comments that you should consider before making your final submission.

  • the relations=[] here is a Python antipattern which makes for bugs: Using a mutable default value as an argument — Python Anti-Patterns documentation

  • You should think about what changes would we need to make to the webservice and add them to your proposal.

  • In general, I get the feeling that we need more details about what endpoints need to be added, what new pages will need to be made. We should make this much more exact.

  • Your UI mockups are good, but some of them give me the feeling of change for change’s sake. You should add more details about your motivation for the UI changes.

wrtiting

  • typo here, you should put this through a spellchecker once. :wink:

  • It is my opinion that taking two weeks for adding one entity is a bit too little for 80 hours of work. Maybe you should consider adding some more work about porting MB ratings to CB somehow, as that is a relatively high priority thing that we’d like to do.

Thanks again for the proposal, don’t forget to submit it to Google! :slight_smile:

@iliekcomputers I have made my final changes. Please review it as if there will be some changes to be done I can do it before the deadline :smiley:

1 Like

Thanks for taking the time to work on the proposal! Don’t forget to submit it to Google before the deadline. :slight_smile:

1 Like

Thanks for your feedback, I have submitted the proposal to Google as well :smiley:

2 Likes