GSOC 2021: Notification Service: Bookbrainz Prabal

Personal information

Name: Prabal Singh
IRC Nick: prabal
MB username: prabalsingh24
Email: praba170107045@iitg.ac.in
Github: prabalsingh24
Time Zone. UTC+05:30

Proposal

This feature is about giving users an option to follow an entity or a collection and get notified whenever there’s any changes made to them

Notification Service

Features

  • A user will be allowed to follow any entity, collection and will be notified whenever there is any changes made to it.
  • We will be having a notification tab in the website which will display all the notifications for the logged in user.
  • Users will also be notified using email. One email per user per day will be sent with all the notification from the previous day.

Frontend Changes

  1. Adding subscribe/unsubscribe buttons in all entity pages and collection page

Database Changes

Whenever there is any edit made in any collection/entity, an entry to the notifications table will be added so that the notification is being shown to the user.

  1. notifications:
    • id
    • subscriber_id
    • read
    • notification_text
    • notification_redirect_link
    • timestamp
CREATE TABLE bookbrainz.notifications (
	id UUID PRIMARY KEY DEFAULT public.uuid_generate_v4(),
	subscriber_id INT,
    read BOOLEAN NOT NULL DEFAULT FALSE,
	notification_text TEXT NOT NULL CHECK (notification_text <> ''),
	notification_redirect_link TEXT NOT NULL CHECK (notification_redirect_link <> ''),
	timestamp TIMESTAMP WITHOUT TIME ZONE NOT NULL DEFAULT timezone('UTC'::TEXT, now()),
	CONSTRAINT notifications_fk FOREIGN KEY (subscriber_id) REFERENCES bookbrainz.editor(id)
);

If a user is following(subscribing) an entity or a collection, they should be notified if there is any change made to it. The information of which user is following which collection or entity will be stored in the ‘entity_subscription’ and ‘collection_subscription’ table

  1. entity_subscription:
    • bbid
    • subscriber_id
CREATE TABLE bookbrainz.enitity_subscriptioin (
	bbid UUID,
    subscriber_id INT,
	CONSTRAINT enitity_subscriptioin_fk FOREIGN KEY (bbid) REFERENCES bookbrainz.entity(bbid),
	CONSTRAINT enitity_subscriptioin_fk1 FOREIGN KEY (subscriber_id) REFERENCES bookbrainz.entity(id)
);
  1. collection_subscription:
    • collection_id
    • subscriber_id
CREATE TABLE bookbrainz.collection_subscription (
	collection_id UUID,
    subscriber_id INT,
	CONSTRAINT collection_subscription_fk FOREIGN KEY (collection_id) REFERENCES bookbrainz.user_collection(collection_id),
	CONSTRAINT collection_subscription_fk1 FOREIGN KEY (subscriber_id) REFERENCES bookbrainz.entity(id)
);

ORM BookBrainz-Data Changes

Three new Models will be added in BookBrainz-data

const Notifications = bookshelf.Model.extend({
                subscriber(){
                       return this.belongsTo('Editor','subscriber_id');
                }
		tableName: 'bookbrainz.notifications'
	});

const EntitySubscription = bookshelf.Model.extend({
                entity() {
			return this.belongsTo('Entity', 'bbid');
		},
         subscriber() {
            return this.belongsTo('Edtitor', 'subscriber_id');
       }
		tableName: 'bookbrainz.entity_subscription'
	});
const CollectionSubscription = bookshelf.Model.extend({
                collection() {
			return this.belongsTo('UserCollection', 'collection_id');
		},
         subscriber() {
            return this.belongsTo('Edtitor', 'subscriber_id');
       }
		tableName: 'bookbrainz.collection_subscription'
	});

Working

A different application - Notification Service - will be created which will handle all the processed related to notification. This code will be written in the bookbrainz repository but this will be run seperately as we do for bookbrainz api.

So whenever there are any changes in the website, website will notify the change to the Notification Service. They both will communicate using RabbitMQ.

RabbitMQ will be setup in both the webserver and in the Notification Service

In the web server Rabbit MQ will be setup like this

Here we will define a function queueNotification which will be used for sending information to notification service

import * as amqp from 'amqplib/callback_api'
let notificationQueue =  'notification_queue'
let queueChannel


amqp.connect(`rabbitMQ-URL`,  (err, connection) => {
     connection.createChannel((err, channel) => {
    	  channel.assertQueue(notificationQueue, { durable: true })
     	  queueChannel = channel
     })
})

export function queueNotifcation(payload) {
  return queueChannel.sendToQueue(notificationQueue, payload), { persistent: true })
}

This function ‘queueNotification’ will be used to send information to the Notification Service.

For example if a user with userId 1000 edits an author with bbid ‘123456’, the following payload will be sent to the Notification Service

const payload = {
	userId: 1000,
	type: entity,
	bbid: ‘123456’,
	entityType: ‘Author’
} 

And this payload will be queued to the queue using **queueNotifcation **function described above

await queueNotification(payload);

This will be done in the entity edit routes (author/ edition/ work/ editionGroup/ publisher edit)

In the same manner if a collection is being edited, it’s payload will be

const payload = {
	userId: 1000,
	type: collection,
	collection_id: ‘123456’,
}

await queueNotification(payload)

This will be done in collection edit route

For the Notification Service, Rabbit MQ will be setup like this


import * as amqp from 'amqplib/callback_api'

let notificationQueue = 'notification_queue'

amqp.connect(`RabbitMQ-URL`, (err, connection: Connection) => {


connection.createChannel((err, channel) => {

channel.assertQueue(notificationQueue);

channel.consume(notificationQueue, async (msg) => {

try {

       const payload = JSON.parse(msg.content.toString())      
       let job
       switch (payload.type) {
          case ‘collection’:
            job = new collectionJob(payload)      // collectionJob is a class described below
            break
          case ‘entity’:
            job = new entityJob(payload)	   // entityJob is a class described below
            break
          default:
            throw new Error("Scenario not declared")
        }
        await job.run()
})

Here entityJob and collectionJob are two classes which will be defined something like this

class collectionJob{
  constructor({userId, collectionId}) {
    this.userId = userId;
    this.collectionId = collectionId;
  }
  run() {
	// the logic of processing this information and making changes in the DB will be written here
 }}
class entityJob{
  constructor({userId, entityId, entityType}) {
    this.userId = userId;
    this.entityId = entityId;
    this.entityType = entityType
  }


  run() {
	// the logic of processing this information and making changes in the DB will be written here
  }
}

The run function in both these classes will be similar and will involve finding out the users that have subscribed that particular entity/collection and then make appropriate change in the notifications table in the DB using ORM models created as mentioned above

Run Algorithm :

Let’s take an example for collection :

let’s assume our payload for this example is

const payload = {
	userId: 1000,
	type: collection,
	collection_id: ‘123456’,
}

Algorithm:
  1. Fetch all the users that have subscribed the collection with collection id 123456 using ORM models (except for when subscriber_id = 1000), let’s say we get subscribers with userId = [1, 2, 3, 4]
  2. Find out the details of the editor with userId = 1000 ( let’s assume editor name = bob) and collection with collection_id = 123456. (let’s assume collection name = harry potter)
  3. Now we have to add the a row in the notifications table. For this our fields would be as described below. We will do this for all the subscribers (1, 2, 3, 4)
{
subscriberId: 1
read: false,
notificationText: 'A collection that you follow - Harry Potter - was edited by Bob',
notificationRedirectLink: '/collection/123456',
timestamp: 
}

This method will create a problem. Let’s say a user edits a collection 20 times, we don’t want their subscribers to get 20 notifications, we just want them to recieve one notification.
The following solution will easily solve this problem:

  1. Before creating a new row in the notifications table, we can check if a similar row exists in the table (with same subscriberId and notificationRedirectLink), if it does, we can just update the time of it and change the read status to false

This also gives us the flexibility to define more notifications scenario.

New Routes in the webserver

In webserver there will be a routes for subscribing and unsubscribing entities/collections

POST /subscribe/entity {
    userId: 1540,
    bbid: ‘asdfasdf’ 
}
POST /unsubscribe/entity{
	userId: ‘1540’,
	bbid: ‘asdfasfd’
}
POST /subscribe/collection {
    userId: 1540,
    collection_Id: ‘asdfsfd’ 
}
POST /unsubscribe/collection {
    userId: ‘1540’,
    collection_id: ‘asdfasfd’
}

The implementation of all these four routes will be similar and it will involve adding or removing a row from the collection_subscription or entity_subscription table.

GET /editor/editorId/notifications?from=0&size=10
This route will be used to display all the notifications of the user. from and size query params will be used for scroller.

POST /notifications/id/state {
     notification_id: '123',
     isRead: true
}

This route will be used to change the state of the notification (read or unread).

POST /notification/id/delete

This route will be used to delete any notification for the user

Notifications using email

There will be a cron job written in Notification Service using node-cron npm package.
This cron job will be run once a day and will send users their notifcations in one email. Notifications can be found by looking at the notifications table in the DB.

TODOs:

  1. Create mock UIs for the frontend part

TIMELINE

Here is a more detailed week-by-week timeline of the GSoC coding period to keep me on track

Whenever any feature/route will be done, I’ll be writing tests using chai and mocha framework for it. From my experience in last year’s GSoC, I learnt that writing tests takes time so it’s best to do it along with the features and not leave it for the end.

Pre Community Bonding Period (April - 17th May) :

This is my last semester in college and and my college ends on 30th April so I won’t be active on the irc in the month of April. But after April, I will be completely free and will be able to work full time on the project

Community Bonding Period :
Because I participated in GSoC 2020, I am familiar with the codebase and I will start coding from the community bonding period.

Week 1 -2 (17th May - 28st May):

I will begin with setting up database (Creating new table) and writing corresponding Models in bookbrainz-data.

Week 3 - 4 (1st June - 11th June):

I will setup subscribe (entity, collection) and unsubscribe (entity, collection) routes and make frontend changes (adding subscribe and unsubscribe buttons).

Week 5 - 8 (14th June - 9th July):

I will setup RabbitMQ in docker and in the webserver in this week.
I will create the Notification Service application, setup rabbitMq, complete the functions as mentioned above.
I will also setup Chai and Mocha framework for testing in the Notification Service

Phase 1 Evaluation here

Week 9 - 10 (12th July - 23rd July):

I will create GET /editor/editorId/notifications endpoint in this week
and will create notifications frontend page.

Week 10 - 11 (26th July - 30th July):

Will create cron job for sending emails to the users in this week

2nd August - 23rd August: Buffer Period

Detailed information about yourself

Tell us about the computer(s) you have available for working on your SoC project!
I have an asus laptop with i7 processor with 8gb of ram.
When did you first start programming?
I wrote Hello World in my computer science class in high school.
What type of books do you read?
I enjoy reading detective novels - Agatha Christie novels are my favourite. I’ve also read Harry Potter, LOTR, The Hunger Games
Have you contributed to other Open Source projects? If so, which projects and can we see some of your code?
I participated in GSOC 2020 and created a feature - UserCollection - for BookBrainz
How much time do you have available, and how would you plan to use it?
I am free during the summer. My college ends in April so from May onwards I’ll be free and will be able to work 30 hours a week.

3 Likes

@mr_monkey have a look at this whenever you’re free

1 Like

Nice proposal so far prabal !

The SQL definitions look sane, although the entity_subscription and collection_subscription definitions have a couple of typos (table name and foreign keys names).
What would the notification_redirect_link be, and what would it be used for?
Could you detail "Appropriate changes in the ORM will be made.” a bit and provide a rough shape of the models?

Where would the notification service live? In the bookbrainz-site repo or separately?
Is there code in the bookbrainz-site repo that would be useful to share, as we do for the api?

Generally, how would the run function of the entityJob and collectionJob access and modify the database? Using the bookbrainz-data-js ORM?
A bit more implementation details here wouldn’t hurt, considering this is the meat of the service and I’m making asumptions.

I don’t think I read that anywhere, but I assume that we would not create notifications for a user if they are the cause of the notification (user_to_notify.id == userId from the notification payload)?

How does the Notification Service keep track of what notifications have been sent/ need to be sent?

I’d like to see some rough draft for how users could save and change their notification preferences, and how that interfaces with the Notification Service.
A good stretch goal would be to allow users to define what emails to receive (and possibly how often), and allowing to disable auto-subscribing when creating entities.

For the timeline, do you think it would take the whole week 4 to set up RMQ? The docker part should be pretty trivial and I don’t envision a lot of issues implementing the plan you’re proposing.
On the other hand, one week to create the new ORM models along with a good series of tests (and reviews) could end up feeling a bit short

Eager to see those todo UI mockups too :slight_smile:

1 Like

I don’t think we need to keep track of these. So basically whenever a user looks up their notifications, it will be fetched from the notifcations table. We’re not sending any notifcations, we’re just storing them in the database and then fetching them whenever required.
As for the notifications sent via email, that will be done once a day and we will send all the notifications from the previous day

I think one week should be enough for this, even last year i was able to do this in one week. But just to be on the safe side, it’s better to have two weeks for this.

I wrote down the algorithm in the proposal. Is that alright?

when notification is clicked, this link would redirect the user to appropriate page . Clicking Bob edited your collection - Harry Potter should redirect user to the collection - Harry Potter

Thanks for those modifications prabal. It’s a bit clearer already.

Here’s a couple more questions:

From your previous message:

As for the notifications sent via email, that will be done once a day and we will send all the notifications from the previous day

I guess what I was wondering about was what happens if either the notifications service or rabbitMQ crash? For the notifications service, I doubt it would be an issue, provided we only acknowledge RMQ messages once we’re sure the email was sent. (otherwise ack RMQ message -> try to send email -> crash -> RMQ message lost !)
For RMQ crashing, let’s consider this out of scope for this project. There is surely a way to persist queues and some best practices around that.

Before creating a new row in the notifications table, we can check if a similar row exists in the table (with same subscriberId and notificationRedirectLink), if it does, we can just update the time of it and change the read status to false

I think that would work and avoid multiple notifications, but it would be better to constrain it by time. For example, only update the time and status of similar notifications from the past 24h.
Otherwise, say I modify a collection, you get a notification, and I modify it the next month, it would be better if you got a new notification. That way the first one is still visible in your notifications.

I see the mockup for the notification bell in the navbar. It’s a bit sparse but I get the idea. I assume there would be different styles if there are unread notifications.
Were you also planning on having a separate page to show and paginate through the notifications?

Not that important, but there are still typos in the SQL definition (“enitity_subscriptioin”).
The second foreign key for both new tables are pointing to the entity table instead of the editor table.
Further down in the ORM definitions there’s also a couple of references to “Edtitor”

I’ll also copy the following paragraph over from my previous message.

“I’d like to see some rough draft for how users could save and change their notification preferences, and how that interfaces with the Notification Service.
A good stretch goal would be to allow users to define what emails to receive (and possibly how often), and allowing to disable auto-subscribing when creating entities.”

I know this year’s project is going to be shorter (so I’m not suggesting it’s a required deliverable) but I’d like you to have a rough idea of how this would be implemented so that throughout the implementation of the project you would be aware of it, so that in the future we don’t have to refactor everything to fit.