GSOC 21 : Add subscriptions and email notifications : Divyanshu

Tags: #<Tag:0x00007f511a64a8a8>

Here is my Draft Proposal for GSOC 21 @BookBrainz on the topic : Add subscriptions and email notifications

The usecase of BookBrainz has been increasing and the graph of number of users is going up day by day, For a user to manage and catchup the edits they make, the entity they create or to follow another artist, user, group, publisher , becomes so much tough and one would feel the need of an assistant to manage those things , though keeping an assistant is not affordable at all :wink: , nevertheless this project aims to provide the ease by contantly reminding the users through email about things they want to follow and manage.
Main Features included :

  • A user wil be notified by default when their is new note on revisons they make or they subscribe
  • A user can also choose to subsribe other user, work, edition, edition-groups, publisher, collection( public ) explicitly
  • Extended goal includes establishing user preference over this default controls.

Frontend changes

Easy access to subscriptions listing

Screenshot from 2021-03-20 16-52-33

  • This would facilate easy and fast access to users subscriptions
  • proper icon will be placed instead of that same as of sigout icon

subscriptions listing

  • can use the idea of collection table to list the subscriptions
  • a checklist and a button to unscribe the multiple subscriptions would be there
  • keeping and removing of table fields are subjected to discussions ahead.
  • pager component will be reused :wink:

subscriber listing

  • will list all the editors( a BB user ) who have subscribed to the edits that we are making
  • listing may also include some specific works of ours that has been been suscribed.
  • For the intial goal only users who are suscribers of us will be displayed in a table
  • pager component will be reused :wink:

Subscribing/UnSuscribing a user

  • A BB user (editor ) can subscribe/Unsuscribe to the other editor by a simple buton interface

  • Likewise a simple button would placed for each of other entity ( TO DO )

    • Work
    • Edition
    • Edition Group
    • Author
    • Publisher
    • Collection

DATABASE Changes

There will be total 8 new table , because we will have 7 relation with editor(user) --> entity, where entities are ( revisions,author, work, editon, editon_group, publisher, collection ) + 1 for user–>user subscription

They will be like :-

user_subscribed_<entity_type>

1 ) user_subscribed_revision

ORM:


CREATE TABLE bookbrainz.user_subscribed_revision(
	user_id serial NOT NULL,
	revision_id uuid NOT NULL,
	CONSTRAINT user_subscribed_author_fk FOREIGN KEY (user_id) REFERENCES bookbrainz.editor(id),
	CONSTRAINT user_subscribed_author_fk_1 FOREIGN KEY (author_id) REFERENCES bookbrainz.revision(id)
);

2 ) user_subscribed_author

Screenshot from 2021-04-03 18-43-00

ORM :

CREATE TABLE bookbrainz.user_subscribed_author (
	user_id serial NOT NULL,
	author_id uuid NOT NULL,
	CONSTRAINT user_subscribed_author_fk FOREIGN KEY (user_id) REFERENCES bookbrainz.editor(id),
	CONSTRAINT user_subscribed_author_fk_1 FOREIGN KEY (author_id) REFERENCES bookbrainz.author_header(bbid)
);

3 ) user_subscribed_work


ORM:

CREATE TABLE bookbrainz.user_subscribed_work (
	user_id serial NOT NULL,
	work_id uuid NOT NULL,
	CONSTRAINT user_subscribed_work_fk FOREIGN KEY (user_id) REFERENCES bookbrainz.editor(id),
	CONSTRAINT user_subscribed_work_fk_1 FOREIGN KEY (work_id) REFERENCES bookbrainz.work_header(bbid)
);

4) user_subscribed_edition


ORM:

CREATE TABLE bookbrainz.user_subscribed_edition (
	user_id serial NOT NULL,
	edition_id uuid NOT NULL,
	CONSTRAINT user_subscribed_edition_fk FOREIGN KEY (user_id) REFERENCES bookbrainz.editor(id),
	CONSTRAINT user_subscribed_edition_fk_1 FOREIGN KEY (edition_id) REFERENCES bookbrainz.edition_header(bbid)
);

5 ) user_subscribed_editon_group

Screenshot from 2021-04-03 19-00-58
ORM:

`CREATE TABLE bookbrainz.user_subscribed_editiongroup (
	user_id serial NOT NULL,
	edition_group_id uuid NULL,
	CONSTRAINT user_subscribed_editiongroup_fk FOREIGN KEY (user_id) REFERENCES bookbrainz.editor(id),
	CONSTRAINT user_subscribed_editiongroup_fk_1 FOREIGN KEY (edition_group_id) REFERENCES bookbrainz.edition_group_header(bbid)
);

6 ) user_subscribed_publisher

Screenshot from 2021-04-03 19-04-31
ORM:

CREATE TABLE bookbrainz.user_subscribed_publisher (
	user_id serial NOT NULL,
	publisher_id uuid NOT NULL,
	CONSTRAINT user_subscribed_publisher_fk FOREIGN KEY (user_id) REFERENCES bookbrainz.editor(id),
	CONSTRAINT user_subscribed_publisher_fk_1 FOREIGN KEY (publisher_id) REFERENCES bookbrainz.publisher_header(bbid)
);

7 ) user_subscribed_collection

CREATE TABLE bookbrainz.user_subscribed_collection (
	user_id serial NOT NULL,
	collection_id uuid NOT NULL,
	CONSTRAINT user_subscribed_collection_fk FOREIGN KEY (user_id) REFERENCES bookbrainz.editor(id),
	CONSTRAINT user_subscribed_collection_fk_1 FOREIGN KEY (collection_id) REFERENCES bookbrainz.user_collection(id)
);

8 ) user_subscribed_user

Screenshot from 2021-04-03 19-21-01
ORM:

CREATE TABLE bookbrainz.user_subscribed_user (
	user_id serial NOT NULL,
	subscriber_id serial NOT NULL,
	CONSTRAINT user_subscribed_user_fk FOREIGN KEY (user_id) REFERENCES bookbrainz.editor(id),
	CONSTRAINT user_subscribed_user_fk_1 FOREIGN KEY (subscriber_id) REFERENCES bookbrainz.editor(id)
);

Setting Up a MAILER

Establishment of A BB dedicated mailer to deliver all our mails to the subscriber is already under progess , almost a step away :slight_smile:
Here i have raised a PR for the same :slight_smile:
It uses :-

  • Nodemailer ( our mailer agent )
  • SMTP server

APIs :

subscribe

  • app.get(’/subscribe/revision/:revisionId’, controller)
  • app.get(’/subscribe/user/:userId’,controller)
  • app.get(’/subscribe/entity_type/:id’,controller) ( will handle user—>entity subscription, entity can be any of 6)

Unsubscribe

  • app.get(’/unsubscribe/revision/:revisionId’, controller)
  • app.get(’/unsubscribe/user/:userId’,controller)
  • app.get(’/unsubscribe/entity_type/:id’,controller) ( will handle user—>entity subscription, entity can be any of 6)

MECHANISM OF MAILING

1) user_subscribed_revision

Suggested by @CatQuest , whenever a user make revision to any entity then they should automatically be added to user_subscribed_revision table and they should recieve email if anyone comments ( add note) on that revision .

  • whenever a user hits the endpoint “/revisions/:id/note” the controller will find the subscribers of that revision from user_subscribed_revision table using revision_id and then send them mail.

2) user_subscribed_entity

Suppose a user subscribes a work entity ,I have assumed following mechanism in order to process subscription :-

  • Any changes made to that entity will count as its revision , so whenever a new revision happens we will seek to the database user_subscribed_work database to find all its subscribers (BB user)
  • We will find what we need to send in email by followig method :-
    • work_revision table contains data_id column
    • using that data_id we will find the final state of work and then
    • Either we can list the changes made to the email body or simply we can use edit-note entered by user while editing the work entity + link of that revision
  • And finally Use the mailer to send the mail to subcribers.

Likewise we will do the same for rest of entity( author, edition, edition_group, publisher, collection).

3) user_subscribed_user

From the discussio so far i have come to conclusion that , any entity modification ( create , delete , revision ) by the user will ultimately be notified to the its subscribers .

Scaling

The best way to scale our system is to scale it horizontaly and establish task-ques to store the jobs to be in bulk and to be processed by separate mailing service.

  • I have a flowchart for the same :-
    Blank diagram

Setting Up RabbitMQ

  • RabbitMQ will be setup using its docker image
  • I have decided to setup this during community bonding period with some guidance of mentors

connection with RabbitMQ

  • Will use ampqlib interface to do so

connect.js

import Interface from 'amqplib' 
let channel  =  Interface.connect(URI)
.then( (connection)=>{ return connection.createChannel() }) 
.catch((error)
=>{ throw error})

export default channel 
// this connection will be utillized by publisher and consumer to  que  and deque the jobs respectively

Publisher

publisher.js

import channel from 'connect.js'
const queName = 'subscriptions'

let job1  = {
email_id:endrance21@gmail.com // users email will be found by looking into `users table`
message:"some message body to be sent",
}


channel.assertQueue(queName)
.then(()=>{
channel.sendToQueue(queName,Buffer.from(JSON.stringify(job1) ,{ persistent: true } ); // will send jobs to the que 
})
.catch(()=>{
//handle error !
})

Consumer

consumer.js

import channel from 'connect.js'
import mailer from "mailer.js"
const queName = 'subscriptions'

channel.assertQueue(queName)
.then(()=>{
return channel.consume(queName); // will consume  jobs to the que 
})
.then((job)=>{
 let {content } = job;
 let body = JSON.parse(content) ; 
let {email,message} = body
mailer.sendMail({email,message}); //this will simply send mail to the given email 
})
.catch(()=>{
//handle error !
})

Preferences :

For now :

  • An user creating any entity wil automatically be subscribed to it using above apis
  • If a user comments or edits any entity then they wil be automatically subscribed to that revision.
  • An user can see others subscriptions and subscribers

Extended goal

Is To provide a prefernce page to user to decide if they want to subscribe to or not to their created entity or revison they make or any comment they make on any revison .

  • user_preference table will be created
    • columns will be decided according to prefernce type but they all will be type of bool .
  • Apis to update the preference
  • app.put(’/preference/update’,controller)
    • Payload will contain the final state of preference table collected through user via checkbox ui
  • app.get(’/preference/,controller)
    • This will fetch the current state of user_preference table and used to display on users profile
  • according to current state of prefernce_table controller will unscribe/subscribe entities .

Detailed information about yourself

Tell us about the computer(s) you have available for working on your SoC project!
A lenevo Legion ( i7, 10 gen + 24GB of RAM) , Separate Monitor along with laptop for multitasking
When did you first start programming?
In My freshmen year of College , i made a Audio Analyser tool using Vanilla JS.
What type of books do you read?
IT ENDS WITH US by COLLEEN HOOVER , NOVEMBER 9 by COLLEEN HOOVER AND IN Non-Fictious I have read ORIGINALS by ADAM GRANT
Have you contributed to other Open Source projects? If so, which projects and can we see some of your code?
I participated in GSOC 2020 as Student Developer , here is the work Product :slight_smile:
How much time do you have available, and how would you plan to use it?
Official GSoC period is from June 7th to 16st August, during the vacation, I will be devoting most of my uptime to this project, 45-50 hrs a week, after re-opening of my college in July, I will be devoting 25-30 hrs a week.

TO DO’s

  • Code snippets
  • Gsoc timeline
  • Error handling and managing enqueing system

Thank You ! :slight_smile:

2 Likes

hey ! @mr_monkey here is my draft proposal.
Please have a look .

1 Like

Thanks for your proposal @endurance21!

I have one main concern with the plan you propose: it’s not going to scale well.
Imagine for example that a lot of editors (think thousands) are subscribed to a popular Author.
Whenever there is a revision added for this Author, the server will have to send thousands of emails. That’s going to slow the server down to a halt and impact the website consequently.
A better approach would be to separate the mailing process somehow, and have a queue of sorts (like RabbitMQ) as an intermediary. Server send a message to RMQ -> RMQ stores a list of jobs that need doing -> notifications system picks messages from RMQ and sends emails.

I don’t know if we’ll need the user_subscribed_revision table, since revisions are already linked to an editor (author_id column).

I’m looking forward to seeing more details, keep it up ! :slight_smile:

2 Likes

I totally agree with this scaling thing and I will update my mechanism accordingly soon.

For the user_subscribed_revision part , let me explain what I thought , suppose an editor A , makes a revision automatically A will be subscribed to that revision means any further note added by any other editor will be notified to A , but if in case another editor B comments on that revision or explicitly subscribe that revision then we would need to maintain that table of all subscribers of revision and of any further note is added to that revision every subscriber will be notified .

Also this was my intial thought , I will amend it accordingly !

@mr_monkey
I have added DATABASE DESIGN and Solution for Scalling this idea.
Code snippets of api implementation and where to put the code is still to do task :slight_smile:
I am on it.

Till then if you get time , it would be helpfull to recieve feedback on above changes :slight_smile:
Thanks !