GSOC 2021 : Add in-site notification system

Personal Information

Name : Anirudh Jain

Nickname : Cyna

IRC nick : Cyna

Musicbrainz userid : anirudhjain75

Email : anirudh.jain@outlook.com

GitHub :https://github.com/anirudhjain75

Twitter :https://twitter.com/anirudhjain75

Proposal

Objective

My objective is to create a notification system for editors in musicbrainz so that they are notified when various events take place on their subscribed entities (i.e. Artists, Albums, Collections, etc) and votes . This is an enhancement over the current notification system which uses emails, to have an additional in-browser notifications management and add i18n to the mix allowing translations of notifications as well. This would prevent email spamming to editors. After the project, the editor would be able to opt on email notifications / in browser notifications or both or none.

Solution

Finalizing the database schema and writing and endpoint to populate the database using SQL queries and data from the perl function called to invoke data changes to the database. the ones created for email notifications would be reused and follow the same convention for further modifications, The formatting of text and translation will be handled before pushing the notification to the notification database.

After completing the API in backend, the task is to move forward with the UI to consume the API /get_notifications created in the previous approach. The parameter would be the editor.id ( used as key identified of ownership of the notifications ).

Creating a basic user interface with a bell icon and overlay of notifications when clicking on it. In addition there would be a notifications page that would contain a list of all notifications received by MusicBrainz for the user. A simple demo can be viewed using the below figma prototype

https://www.figma.com/file/HVBTB4PAk7MBkfEqhkk9Kc/MusicbrainzNotifications?node-id=0%3A1

Clicking on the notification icon ( bell icon ) reveals a small overlay to display short list notifications. And clicking on Show All in minor preview would display the notifications page coupled along with other user attributes. This notifications page can be used to track all the previous notifications / alerts generated that are not visible on the notifications preview.

In addition to adding notification systems to browser level. Translated notifications will also be possible using i18n currently followed in musicbrainz website improving upon email notifications which do not support translated notifications.

Notification Schema that will be used and referred for the project

  • id [SERIAL] PK
  • receiver_id ( editor.id ) [INTEGER]
  • content [TEXT]
  • datetime_added [TIMESTAMP WITH TIME ZONE]
  • datetime_read [BOOLEAN]
  • sender_id [INTEGER]

Types of Notifications currently supported by Musicbrainz Project

  • Changes in Subscribed Entities
  • Changes by Subscribed editor
  • Deletion of Subscribed Entity

Details on Each Notifications and their shorthands to be used

New Releases - Not used at the moment

Change format to limit 1 release per notification for clearer message. Email sends bulk of releases as single notification

Shorthand : New Release - Blood of Cromos by Iron Maiden

Long : Blood of Cromos by Iron Maiden released on 10th feb 2077 on Spotify

ClickableLink : Points to release with name - Blood of Cromos by Iron Maiden

Changes to Subscribed Entities ( artist, collection, labels, series ) - admin/ProcessSubscriptions

Shorthand : Edits made to Iron Maiden Artist

Long : Edits made to Iron Maiden Artist by $editorName

ClickableLink : $ ServerURL /artist/ $IronMaidenGid /edits

Shorthand : Edits made to Blood of Cromos collection

Long : Edits made to Blood of Cromos collection by $editorName

ClickableLink : $ ServerURL /artist/ $IronMaidenGid /edits

Deletion of Subscribed Entities - admin/ProcessSubscriptions

Shorthand : Blood of Cromos has been deleted or merged( Collection )

Long : Some of your subscribed artists, labels or collections have been merged, deleted or made private

Clickable : $ServerURL /edit/ $EditIdentifier

Changes by Subscribed Editor - admin/ProcessSubscriptions

Shorthand : Changes made by yvanzo

Long : Changes made by yvanzo

ClickableLink : $ServerURL /user/ yvanzo /edits

AutoEditorElections - Hourly.sh

ClickableLink - /election/electionId

Accepted Edit

Shorthand : $ CandidateName has been accepted as an auto editor

Long : Voting in this election is now closed: $ CandidateName has been

accepted as an auto-editor. Congratulations! For more details, click here

Cancelled Edit

Shorthand : Election has been cancelled by Proposer ( $ProposerName )

Long : This election has been cancelled by the proposer ( $ProposerName ). For more details, click here

Nominated Edit

Shorthand : A new candidate has been put forward for auto editor status

Long : A new candidate has been put forward for autoeditor status:

Candidate: $CandidateName

Proposer: $ProposerName

Rejected Edit

Shorthand : The proposal to make $CandidateName an auto editor was declined

Long : Voting in this election is now closed: the proposal to make $CandidateName an auto-editor was declined. For more details, click here,

Timeout Edit

Shorthand : Elections have been cancelled due to timeout

Long : This election has been cancelled, because two seconders could not be found within the allowed time. For more details, click here

Voting open for Edit

Shorthand : Voting ot elect auto editor is now open

Long : Voting to elect $CandidateName as auto editor has been proposed by $ProposerName. For more details, click here.

The Project can be split into 3 phases.

  • Writing SQL queries and modifying schema
  • Creating Web API in Perl to be consumed by the web interface and the backend interacts with notifications table.
  • Creation of Web Interface

Writing SQL queries and Modifying schema.

SQL queries need to be written for two operations. Adding notification to Notifications bank of a user ( requested and performed by the server ). Fetching notifications of a user ( requested from client and performed by the server ). Reference to schema can be found in the above topic.

Creating Web API in Perl to be used by the web interface.

Web API endpoints have to be created to invoke and get the json response of a list of notifications of a user. The endpoint /get_notifications which would earlier be handled by backend would make things dynamic and work with the database.

Creation of Web Interface

This contains adding notification icon, notification preview and notifications page. The test would be done using API created to fetch notifications.

How are notifications being populated for a user ?

Notifications are being handled by cron jobs running all the time on musicbrainz server. Daily, Hourly and Weekly.

Hourly cron jobs run to populate / create mail for users compiling the list of election results and votes the editor is interested in.

Daily / Weekly cron jobs are used for notifying users on entities they are subscribed to

Later simple addition to allow p2p messaging can be made by using instantaneous / queued messaging for heavy load. New functions to modify data to add messages to the database will be created to push notification at the moment they are made

How will they be added to Musicbrainz current system ?

All notifications generated on mail are based on functions and I will interject code to add the same notification by making some change to the format to the database. This would serve as the source for /getNotification API.

How will translations be handled
Three plan of implementing translations are possible.

  1. Let UI handle the notifications translation using l, lp tag as used by other components. Drawbacks are the transifex will be called many times for a single editor for the same language.

  2. Let backend handle the notification translation by translation on trigger and store in database in translated format. Drawback is that if editor changes the language, there would be no simple way to translate the context as they would be translated and stored in database, this can be worked on by storing multiple languages of notification when requested by adding language parameter to schema. which might effect the storage size of the database.

  3. Let UI handle the translation but use json file with i18n to handle the translation on UI end. Any additional text would have to be translated and updated in the json file. For more info on the module that can be used. https://www.npmjs.com/package/i18n-react

I will be going with first approach as per the mentor’s suggestion. The solution is so simply use the existing way on how things are translated in musicbrainz website. ( using l, lp tags in javascript ). The use of l, lp automatically translated the entire content based on existing templates in Transifix. An example could be seen at musicbrainz-server/root/artist/ArtistIndex.js on how L tag is used. Translations can be found in .po files in musicbrainz-server/po.

Currently I have some confusions on how I’ll add more strings to be translated to the existing translation system. Although the l tag can be used both with and without parameter therefore adding links and other replacements are possible.
Usage :

l('You may proceed to <a href=\"{url}\">your user profile</a>,
      or maybe you want to {edit_link}', url => c.uri_for('/user/profile
     '), edit_link => doc_link("How_Editing_Works", l("start editing!"))

Somewhere around the lines of this would be written for all the statements and then added so in theory the translations should work with the currently selected language as its done already by the website.

Implementation timeline

7th June - 13th June : Finalize the proposed schema and complete the backend API /get_notifications

14th June - 20th June : Handing the test cases for the API ensuring its working

21st June - 27th June : Inject code in functions of hourly, daily and weekly cron jobs to populate notifications database based on schema and refactor based on above mentioned strings.

28th June - 4th July : Adding test cases if any, Finalize the working on the backend and the API to progress with the UI

5th July - 11th July : Complete a basic UI as proposed and add setting to toggle in browser notification

12th July - 16th July - Phase 1 Evaluations : Basic UI with backend and API completed

17th July - 23rd July : Work on adding translations to the mix and investigate p2p messaging using notifications approach.

24th July - 30th July : Create functions for sending instantaneous messages instead of cron jobs for p2p messaging and other required alerts.

31st July - 6th August : Write test cases if any, Ensure the work on p2p messaging / instantaneous notifications are complete.

7th August - 15th August : Buffer period for any pending backlogs or improvements suggested on real time

16th August - 23rd August : Push the remaining code for review and finalize the completion of project

Other Information

1. Tell us about the computer(s) you have available for working on your SoC project!

I have macbook pro 2015 with 16 gb ram. And a custom built pc with ryzen 5600x, 16gb and rtx 3070.

2. When did you first start programming?

During High School - 2017

3. What type of music do you listen to? (Please list a series of MBIDs as examples.)

Periphery

Nothing Else Matters - Metallica

Stairway to Heaven - Led Zeppellin

4. What aspects of the project you're applying for (e.g., MusicBrainz, AcousticBrainz, etc.) interest you the most?

Notification system being one of the most basic feature yet highly required by most sites for an interactive experience with users. The aspect of creating it for a majorly released website is in itself a good opportunity as well as the research and reason behind what is the best way to implement it is a good architectural challenge for me, which obviously I am interested the most to solve.

5. Have you ever used MusicBrainz to tag your files?

I haven't used it yet and hope to use it in future if I actually have music files stored on my PC. I usually use online music services such as Spotify, Apple Music and Youtube Music.

6. Have you contributed to other Open Source projects? If so, which projects and can we see some of your code?

Yes I have contributed to Layer5's Meshery project as well as Musicbrainz project of Metabrainz Foundation.

7. What sorts of programming projects have you done on your own time?

I have done a couple of programming projects, Websites with Accessibility options ( Real time gestures recognition, Voice based Navigation, Various color effects for partial blindness aid ), Creating Game Servers using Kubernetes Manifest ( I've made one for Ark Survival Evolved ), Writing API server using API Gateway and AWS lambda for Voice Assistant ( in my case, Primarily to build skill for Amazon Alexa ). And a lot more.

8. How much time do you have available, and how would you plan to use it?

I have about 4-5h a day allocated for Summer of Code. I plan on learning anything I need to complete the project ( in my case, Perl with MOOSE ) and implementing it. I plan on wrapping the project early to work on enhancements to the notification system.

3 Likes

Thank you for your detailed proposition of a long-awaited feature by MusicBrainz editors.

Your prototype UI proposes very basic features only but that is better than proposing a fancy UI that could never be realistically implemented within the 150h of GSoC. As you already participated to GSoC it is expected you are able to better manage your time, so this is a good start. :slight_smile:

About your proposed SQL table

editor.id is of type SERIAL. See source code.

There is no datatype “url string“ in Postgres. url.url just uses TEXT instead. See source code.
What is the purpose of this column? I guess the link can just be added to the notification content.

Other columns make sense.

Implementation detail: Column names should follow the current naming convention: separate each word with an underscore like_that, rather than using CamelCase. See source code.

About pulling notifications

Your proposition of creating a dedicated web API endpoint makes sense. How do you plan to use Mock API with MB server? Did you use it before? Otherwise, I would suggest you to take a look at our mechanized tests instead. See source code.

About pushing notifications

The main reason notification emails are sent only once a day is not to spam editors, for both their own mailboxes and for our mail sending quotas. This makes processing notifications to take a quite large amount of resources and time. In-site notifications do not have this problem, so would it be possible to update the notification table more frequently and would it be faster?

Conclusion

After updating your proposition’s content, please try to add more details to your timeline too. I don’t really understand how you would be developing the UI before the back-end, so maybe reorder these more logically. Last, i18n will be a nice improvement over email notifications but it does not need a dedicated week, it is just about using l, lp, ln functions.

1 Like

Hi Yvanzo, Thanks for reviewing the proposal. I have made the suggesting changes on the schema.

Regarding

Your proposition of creating a dedicated web API endpoint makes sense. How do you plan to use Mock API with MB server? Did you use it before?

I dont plan on integrating it with the mb server, It would simply be an external endpoint pinged from the website. The reason here is to complete the items that can be completed and left aside so work on backend part can proceed easily using the mock API as reference. and Yes I have created mock API servers before using both Postman and writing a server using Go for prototyping.

The main reason notification emails are sent only once a day is not to spam editors, for both their own mailboxes and for our mail sending quotas. This makes processing notifications to take a quite large amount of resources and time. In-site notifications do not have this problem, so would it be possible to update the notification table more frequently and would it be faster?

Yes the notification system can be enhanced upon to directly introduce a generalized alert function ( to be called while an entity is updated / created ) which can be called to send notification to the subscribed editors database. The current approach is to set the foundation for insite realtime notifications and messaging which is what we want in the end I presume.

After updating your proposition’s content, please try to add more details to your timeline too. I don’t really understand how you would be developing the UI before the back-end, so maybe reorder these more logically. Last, i18n will be a nice improvement over email notifications but it does not need a dedicated week, it is just about using l, lp, ln functions.

Sure I’ll work on adding more detail to the timeline and complete the pending UI portion. I will develop UI before backend using the Mock API approach. As for i18n, I gave that a week to ensure its completion in any case I get stuck. We can discuss more on what time can be spent on to improve the proposal. :slight_smile:

Thanks for Reviewing,
Anirudh Jain ( Cyna )

Tests that serve for reference must be automated and placed under t/. So if you want to use Mock API as a reference, you need either to integrate it under t/or to write additional mechanized tests under t/. In both cases, that seem to be extra effort. You can still use Mock API if you feel more comfortable with, but we cannot help you with and it does not replace having back-end tests of some sort under t/.

It takes a few minutes, or at worst a few hours if you deal with hydrated components.

Thanks for the proposal, Cyna!

While you’re free to make a proposal focusing only on notifications, the primary feature we focused on in our suggestion was “a way to communicate with other editors without sending an email, and to receive notifications when people comment on your edits without getting your email inbox flooded.”

The reason we tied those together is that we envisioned a “notification” would simply be a message (like one sent between editors) from a system user that you can’t reply to. It would be nice to start from that use case when designing the schema so that we don’t have to make changes later. (It’s possible that you disagree that messages between editors and notifications should share their implementation like this, but I don’t see any mention of the former in the proposal in either case.)

So I think the proposal and schema need some work to address this. But it’s not a significant rework.

I imagine the message will usually contain links, so if the notification itself is clickable, it’d simply expand the message so that you can click on the links. So this column doesn’t appear to be useful for us.

A complete implementation of the “get notifications” API should be very simple – probably simple enough that you can include the SQL in the proposal. So I’m skeptical of starting with a Mock API. I’d suggest starting on the schema and backend first, because these are the parts that we’d want to get 100% right first.

I think the tricky part is that we’d want the messages translated according to the UI language, and the messages usually need some variables interpolated into them. So we have to store the message to translate plus the variables. I was hoping the proposal would discuss this a bit more, because we do something similar for ModBot notes and they could share an implementation somehow: https://github.com/metabrainz/musicbrainz-server/blob/master/lib/MusicBrainz/Server/Entity/EditNote.pm#L57

(Edit: But I agree i18n doesn’t need a dedicated week.)

1 Like

Thanks for the review bitmap. I’ll refactor the proposal and add some stuff to it. As for the ability to allow sending messages from editor to editor. You said in the IRC that thats not the main focus so didnt mention that in the proposal. I initially thought of adding ability to add feature for editor to send message to other editor by using the sender id in the notifications database.

As for the translations, I’ll be using the language set in musicbrainz site using l, lp, etc. Will add more details soon. Hope to submit before the deadline

Hi @Bitmap @yvanzo @reosarevok I’ve made some changes to the proposal making it better I hope. Please do review it and suggest improvements if possible. Last week of the GSoC timeline is marked as buffer as the project is planned to be completed beforehand.

Additional images of UI and prototype will follow as there are some design considerations I am making.

1 Like

Sorry if something I said was unclear, but I don’t think I said that. The only relevant IRC chat I could find was from 2021-03-12, and it matches what I said here:

[18:18:20] <Cyna[m]> Yea.. also we are looking for P2P messages between editors as well right ?.
[18:18:40] <Cyna[m]> I mean it was part of the notifications jira I looked into
[18:21:31] <bitmap> yes, I think ideally a notification would just be a message (from a system user) that you can't reply to
[18:22:07] <bitmap> at least that would be enough to cover the functionality we want
[18:25:44] <Cyna[m]> Ok

The suggestion at https://wiki.musicbrainz.org/Development/Summer_of_Code/2021/MusicBrainz also mentions “messaging” or “messages” equally or more times than it does “notification,” with “a way to communicate with other editors without sending an email” as a primary goal in the first sentence, so I’m not sure where you got this idea that it wasn’t a main focus. :frowning:

I’d still recommend that you tweak your proposal to more closely follow the objectives outlined in the suggestion we gave.

Using “trigger” in the same sentence as database schema and SQL implies https://www.postgresql.org/docs/12/triggers.html to me, but I don’t think that’s what you mean, since we don’t use database triggers for email notifications. So I’d just clarify what you mean by “the trigger” here and avoid that word if possible. :slight_smile:

Two of the data types here aren’t right. (I’ll let you figure which ones. :))

Have a look at the column naming conventions we use in the rest of our schema. These names don’t match our conventions at all.

Like I said, try to keep the schema based on inter-user messaging from the start, not “Incase of implementing p2p messaging between users.”

A boolean should suffice for datetime_read, since it’s only for the receiver to know what messages they’ve read.

I’d remove any mention of “New Releases” from the proposal, since it’s not a feature we support.

The “AutoEditorElections” section needs fixing. For one, I don’t think we need to support auto-editor notifications as part of the GSoC project, since we currently have to email those manually (unless you plan to look into fixing that).

The other problem with this section is that you’re mixing it up with edit notifications (e.g. “Nominated Edit” doesn’t make any sense).

I’m not sure what “using instantaneous / queued messaging for heavy load” means in this context. The implementation should be the same, and messaging shouldn’t cause heavy load.

Have you looked at how we translate edit notes left by ModBot? I pointed you to that in IRC in March and in this thread above.

Nitpick: Transifex is the localization platform we use, but we don’t call it at runtime. We pull .po files from there into the repository. For translation at runtime we use https://messageformat.github.io/Jed/

This isn’t a realistic option, so I don’t see a need to mention it in the proposal.

This ultimately isn’t that different from option 1, since .po files are converted to JSON for Jed already. The problem here is that i18n-react isn’t based on gettext, so it’s not something we could use (and there’s nothing it provides that we can’t do with Jed already, other than Markdown syntax, I guess).

To be clear, we’d like notifications to be based on messaging (as in notifications are a type of message), not the other way around.

I’m not sure what this task involves. Like above, it might be that the word “trigger” is throwing me off.

Something the proposal doesn’t really discuss is how the UI for messaging would work. Can we reuse the existing “Send Email” page?