GSOC 2021: Series Entity: BookBrainz Akash

Personal information

Name: Akash Gupta

IRC Nick: akashgp09

Email: akashgp9@gmail.com

Github: akashgp09

Portfolio: akashgp09.co

Proposal

Project overview

Bookbrainz lacks a feature which allows a person to create a series of entities. A series is a sequence of separate author, work, edition, edition-group with a common theme. The individual entities will often have been given a number indicating the position in the series.

Features

Basic Features :

  • A user can create a series entity of a particular entity type.For ex: Author-Series, Work-Series, Edition-Series,Edition-Group-Series and can store the entities of that type in the Series.

  • Implement a type Property. The type primarily describes what type of entity the series contains. The type property will restrict the series entity to just one entity type. No series entity will hold multiple entity(i.e work and edition within a same series entity is invalid)

  • Implement a property Ordering type, which will provide the user a choice to order their individual entities in the series manually or automatically.

  • Manual Ordering of entities in series provides flexibility to the user to self position their individual entities by assigning a number to each entity indicating their order in the series.

UI Prototype :

The add navbar item on the homepage will now have an additional item: Series.

Note: For Better presentation of UI I have arbitrarily picked an icon to represent the series entity. It will be changed accordingly

Clicking on the Series menu item will take us to Series Entity Creation page

Similar to other entities, a user can enter the Name, Sort Name, Language, Disambiguation, Alias and Identifiers for the series entity.

The above Type input will allow users to choose the entity type for the series entity from a list of options.

(Note: For now we have just considered only 4 entity types : Author Series, Work Series, Edition Series and Edition Group Series. According to our use case we can remodel it accordingly)

The Ordering Type input will have two options: Automatic and Manual.

The Add Relationship Modal will now have an extra input field to assign a position to the entity in the series

Manual Ordering of Entities :

This will provide users the ability to position the individual entities and display them in the order they want by assigning a number to each entity indicating their position in the series.

Example of a Work Series Entity Ordered manually :

A column labelled Order indicates the position on the individual entities displayed.

Automatic Ordering of Entities :

When a user selects the ordering type as automatic, the individual entity will be displayed in the order they were added in the Series. Moreover there will be no Order Column in the series entity table.

Example of a Work Series Entity Ordered Automatically :

The Series Entity Shown above is an example of type Work Series. Depending on the type of series, The Series entity will render the respective entity-table .

For Example : Series Entity of Type Work will render work-table component and Series Entity of Type Edition will render edition-table component.

Database Changes

Five new tables will be added in the existing database :

  • series_data
  • series_header
  • series_revision
  • series_type
  • series_ordering_type

  • series-type: this table will store the types that a series entity can have. (for example type: work series, edition series etc…). The type_id column in the series_data accomplishes the job of fetching and restricting the type of entities in a series.

  • Series_ordering_type: this table will contain the ordering type. which determines whether the series is ordered automatically or manually. The ordering_type column in series_data table is responsible for restricting the ordering type of a Series Entity.

  • The other three tables i.e series_data, series_revision and series_header for series entity will be similar to edition_data/edition_group_data, edition_revision/edition_group_revision and edition_header/edition_group_header tables of edition and edition_group entities respectively.

CREATE TABLE bookbrainz.series_type (
id SERIAL PRIMARY KEY,
label TEXT NOT NULL UNIQUE CHECK (label <> '')
);
CREATE TABLE bookbrainz.series_ordering_type(
id SERIAL PRIMARY KEY,
label TEXT NOT NULL UNIQUE CHECK (label <> '')
);
CREATE TABLE bookbrainz.series_data (
id SERIAL PRIMARY KEY,
alias_set_id INT NOT NULL REFERENCES alias_set(id),
identifier_set_id INT REFERENCES bookbrainz.identifier_set(id),
relationship_set_id INT REFERENCES bookbrainz.relationship_set(id),
annotation_id INT REFERENCES bookbrainz.annotation(id),
disambiguation_id INT REFERENCES bookbrainz.disambiguation(id),
language_set_id INT REFERENCES bookbrainz.language_set(id),
type_id INT REFERENCES bookbrainz.series_type (id),
ordering_id INT REFERENCES bookbrainz.series_ordering_type(id)
);
CREATE TABLE bookbrainz.series_header (
bbid UUID PRIMARY KEY,
master_revision_id INT
);
CREATE TABLE bookbrainz.series_revision (
id INT REFERENCES bookbrainz.revision (id),
bbid UUID REFERENCES bookbrainz.series_header (bbid),
data_id INT REFERENCES bookbrainz.series_data(id),
is_merge BOOLEAN NOT NULL DEFAULT FALSE,
PRIMARY KEY ( id, bbid )
);
ALTER TABLE bookbrainz.series_header ADD FOREIGN KEY (bbid) REFERENCES bookbrainz.entity (bbid);
ALTER TABLE bookbrainz.series_header ADD FOREIGN KEY (master_revision_id, bbid) REFERENCES bookbrainz.series_revision (id, bbid);

  • One Additional Change to the existing database will be adding a position column to the relationship table.
ALTER TABLE bookbrainz.relationship ADD COLUMN position INT;

/∗ Sample view for series ∗/

CREATE VIEW bookbrainz.series AS
SELECT
e.bbid, sd.id AS data_id, sr.id AS revision_id, (sr.id = s.master_revision_id) AS master, sd.annotation_id, sd.disambiguation_id,
als.default_alias_id, sd.type_id, sd.ordering_id, sd.alias_set_id, sd.language_set_id, sd.identifier_set_id,
sd.relationship_set_id, e.type
FROM bookbrainz.series_revision sr
LEFT JOIN bookbrainz.entity e ON e.bbid = sr.bbid
LEFT JOIN bookbrainz.series_header s ON s.bbid = e.bbid
LEFT JOIN bookbrainz.series_data sd ON sr.data_id = sd.id
LEFT JOIN bookbrainz.alias_set als ON sd.alias_set_id = als.id
WHERE e.type = 'Series';

ORM BookBrainz-Data Changes

Four new models will be added in BookBrainz-data

series, series-data, series-revision and series-header

  • series-data :
import { camelToSnake, snakeToCamel } from "../../util";

export default function seriesData(bookshelf) {
	const SeriesData = bookshelf.Model.extend({
		aliasSet() {
			return this.belongsTo("AliasSet", "alias_set_id");
		},
		annotation() {
			return this.belongsTo("Annotation", "annotation_id");
		},
		disambiguation() {
			return this.belongsTo("Disambiguation", "disambiguation_id");
		},
		seriesType() {
			return this.belongsTo("SeriesType", "type_id");
		},
		seriesOrderingType() {
			return this.belongsTo("SeriesOrderingType", "ordering_id");
		},
		format: camelToSnake,
		idAttribute: "id",
		identifierSet() {
			return this.belongsTo("IdentifierSet", "identifier_set_id");
		},
		languageSet() {
			return this.belongsTo("LanguageSet", "language_set_id");
		},
		parse: snakeToCamel,
		relationshipSet() {
			return this.belongsTo("RelationshipSet", "relationship_set_id");
		},
		tableName: "bookbrainz.series_data",
	});

	return bookshelf.model("SeriesData", SeriesData);
}

  • series
export default function series(bookshelf) {
	const SeriesData = bookshelf.model("SeriesData");
	const Series = SeriesData.extend({
		defaultAlias() {
			return this.belongsTo("Alias", "default_alias_id");
		},
		idAttribute: "bbid",
		initialize() {
			this.on("fetching", (model, col, options) => {
				// If no revision is specified, fetch the master revision
				if (!model.get("revisionId")) {
					options.query.where({ master: true });
				}
			});

			this.on("updating", (model, attrs, options) => {
				// Always update the master revision.
				options.query.where({ master: true });
			});
		},
		revision() {
			return this.belongsTo("SeriesRevision", "revision_id");
		},
		tableName: "bookbrainz.series",
	});

	return bookshelf.model("Series", Series);
}

  • series-revision:
import { camelToSnake, diffRevisions, snakeToCamel } from "../../util";

export default function seriesRevision(bookshelf) {
	const SeriesRevision = bookshelf.Model.extend({
		data() {
			return this.belongsTo("SeriesData", "data_id");
		},
		diff(other) {
			return diffRevisions(this, other, [
				"annotation",
				"disambiguation",
				"aliasSet.aliases.language",
				"aliasSet.defaultAlias",
				"relationshipSet.relationships",
				"relationshipSet.relationships.type",
				"seriesType",
				"seriesOrderingType",
				"languageSet.languages",
				"identifierSet.identifiers.type",
			]);
		},
		entity() {
			return this.belongsTo("SeriesHeader", "bbid");
		},
		format: camelToSnake,
		idAttribute: "id",
		parent() {
			return this.related("revision")
				.fetch()
				.then((revision) =>
					revision.related("parents").fetch({ require: false })
				)
				.then((parents) => parents.map((parent) => parent.get("id")))
				.then((parentIds) => {
					if (parentIds.length === 0) {
						return null;
					}

					return new SeriesRevision()
						.where("bbid", this.get("bbid"))
						.query("whereIn", "id", parentIds)
						.orderBy("id", "DESC")
						.fetch();
				});
		},
		parse: snakeToCamel,
		revision() {
			return this.belongsTo("Revision", "id");
		},
		tableName: "bookbrainz.series_revision",
	});

	return bookshelf.model("SeriesRevision", SeriesRevision);
}
  • series-header
import { camelToSnake, snakeToCamel } from "../../util";

export default function seriesHeader(bookshelf) {
	const SeriesHeader = bookshelf.Model.extend({
		format: camelToSnake,
		idAttribute: "bbid",
		parse: snakeToCamel,
		tableName: "bookbrainz.series_header",
	});

	return bookshelf.model("SeriesHeader", SeriesHeader);
}


Backend Server Changes

I will add two new middlewares in src/server/helpers/middlewares :

  • loadSeriesTypes
  • loadSeriesOrderingTypes

export const loadSeriesTypes= makeLoader('SeriesType','seriesTypes');

export const loadSeriesOrderingTypes= makeLoader('SeriesOrderingType','seriesOrderingTypes');

In file routes/entity.tsx we will need to modify our constructRelationships( ) function by adding the position property newly added to relationship table.


export function constructRelationships(relationshipSection) {
	return _.map(
		relationshipSection.relationships,
		({rowID, relationshipType, sourceEntity, targetEntity, position}) => ({
			id: rowID,
			sourceBbid: _.get(sourceEntity, 'bbid'),
			targetBbid: _.get(targetEntity, 'bbid'),
			typeId: relationshipType.id,
			position
		})
	);
}

Our series entity will have similar routes like the other entities we have:

We will create a new file for our series entity routes which will have the following routes:

In src/server/routes/series.js:

function transformNewForm(data) {
	const aliases = entityRoutes.constructAliases(
		data.aliasEditor,
		data.nameSection
	);

	const identifiers = entityRoutes.constructIdentifiers(
		data.identifierEditor
	);

	const relationships = entityRoutes.constructRelationships(
		data.relationshipSection
	);

	return {
		aliases,
		annotation: data.annotationSection.content,
		disambiguation: data.nameSection.disambiguation,
		identifiers,
		note: data.submissionSection.note,
		relationships,
		typeId: data.seriesSection.type,
	};
}

const createOrEditHandler = makeEntityCreateOrEditHandler(
	"series",
	transformNewForm,
	"typeId"
);

const mergeHandler = makeEntityCreateOrEditHandler(
	"series",
	transformNewForm,
	"typeId",
	true
);

const router = express.Router();

// Creation
router.get(
	"/create",
	auth.isAuthenticated,
	middleware.loadIdentifierTypes,
	middleware.loadLanguages,
	middleware.loadSeriesTypes,
	middleware.loadSeriesOrderingTypes.middleware.loadRelationshipTypes,
	(req, res) => {
		const { markup, props } = entityEditorMarkup(
			generateEntityProps("series", req, res, {})
		);

		return res.send(
			target({
				markup,
				props: escapeProps(props),
				script: "/js/entity-editor.js",
				title: props.heading,
			})
		);
	}
);

router.post(
	"/create/handler",
	auth.isAuthenticatedForHandler,
	createOrEditHandler
);

router.param("bbid", middleware.redirectedBbid);

router.param(
	"bbid",
	middleware.makeEntityLoader(
		"Series",
		[
			"seriesType",
			"series.defaultAlias",
			"series.disambiguation",
			"series.identifierSet.identifiers.type",
			"seriesOrderingType",
		],
		"Series not found"
	)
);

function _setSeriesTitle(res) {
	res.locals.title = utils.createEntityPageTitle(
		res.locals.entity,
		"Series",
		utils.template`Series “${"name"}”`
	);
}

router.get("/:bbid", middleware.loadEntityRelationships, (req, res) => {
	_setSeriesTitle(res);
	res.locals.entity.series.sort(entityRoutes.compareEntitiesByDate);
	entityRoutes.displayEntity(req, res);
});

router.get("/:bbid/delete", auth.isAuthenticated, (req, res) => {
	_setSeriesTitle(res);
	entityRoutes.displayDeleteEntity(req, res);
});

router.post(
	"/:bbid/delete/handler",
	auth.isAuthenticatedForHandler,
	(req, res) => {
		const { orm } = req.app.locals;
		const { SeriesHeader, SeriesRevision } = orm;
		return entityRoutes.handleDelete(
			orm,
			req,
			res,
			SeriesHeader,
			SeriesRevision
		);
	}
);

router.get("/:bbid/revisions", (req, res, next) => {
	const { SeriesRevision } = req.app.locals.orm;
	_setSeriesTitle(res);
	entityRoutes.displayRevisions(req, res, next, SeriesRevision);
});

router.get("/:bbid/revisions/revisions", (req, res, next) => {
	const { SeriesRevision } = req.app.locals.orm;

	_setSeriesTitle(res);
	entityRoutes.updateDisplayedRevisions(req, res, next, SeriesRevision);
});

router.get(
	"/:bbid/edit",
	auth.isAuthenticated,
	middleware.loadIdentifierTypes,
	middleware.loadSeriesTypes,
	middleware.loadSeriesOrderingTypes,
	middleware.loadLanguages,
	middleware.loadEntityRelationships,
	middleware.loadRelationshipTypes,
	(req, res) => {
		const { markup, props } = entityEditorMarkup(
			generateEntityProps("seriesGroup", req, res, {}, seriesToFormState)
		);

		return res.send(
			target({
				markup,
				props: escapeProps(props),
				script: "/js/entity-editor.js",
				title: props.heading,
			})
		);
	}
);

router.post(
	"/:bbid/edit/handler",
	auth.isAuthenticatedForHandler,
	createOrEditHandler
);

router.post(
	"/:bbid/merge/handler",
	auth.isAuthenticatedForHandler,
	mergeHandler
);

export default router;


We will add a new function getSeriesRelationshipTargetByTypeId in src/client/helpers/entity.tsx


export function getSeriesRelationshipTargetByTypeId(
	entity,
	relationshipTypeId: number
) {
	let targets = [];
	if (Array.isArray(entity.relationships)) {
		targets = entity.relationships.filter(
			(relation) => relation.typeId === relationshipTypeId
		);
		if (targets[0].postion) {
			targets.sort(function (a, b) {
				return a.position - b.postion;
			});
		}
		targets.map((relation) => {
			const { target } = relation;
			return target;
		});
		return targets;
	}
}

This function is similar to getRelationshipTargetByTypeId. The only noticeable difference is that it will check if the relationship object has truthy position value. Having a position value indicates the series entity is ordered manually and having a position value as null indicates the series entity is ordered automatically and we don’t require sorting operation to perform before displaying it.

Suppose we have a relationships array as :

relationships: (3) [ { … } , { … } , { … } ]

[
    id: 289,
    typeId: 10,  (typeId: 10 for series entity of type work)
    sourceBbid: "...",
    targetBbid: "...",
    position: 3,
    target: {
        ...
    },
    id: 290,
    typeId: 10,
    sourceBbid: "...",
    targetBbid: "...",
    position: 1,
    target: {
        ...
    },
    id: 291,
    typeId: 10,
    sourceBbid: "...",
    targetBbid: "...",
    position: 2,
    target: {
        ...
    }
]

The targets.sort array iterator will sort the above array of objects since we got a truthy position value. The sorted result will be :


[
    id: 290,
    typeId: 10,
    sourceBbid: "...",
    targetBbid: "...",
    position: 1,
    target: {
        ...
    },
    id: 291,
    typeId: 10,
    sourceBbid: "...",
    targetBbid: "...",
    position: 2,
    target: {
        ...
    },
    id: 289,
    typeId: 10,
    sourceBbid: "...",
    targetBbid: "...",
    position: 3,
    target: {
        ...
    }
]

After sorting the array we can extract the target property from the array of objects and pass it to the client side just like what getRelationshipTargetByTypeId does.

TIMELINE

Here is a more detailed week-by-week timeline of the 10 weeks GSoC coding period to keep me on track

Pre-Community Bonding Period ( April ) :

I believe, to accomplish the job of implementing a series entity, one should have a good understanding with the implementation of other entities. Almost more than 70% of the implementation of the series entity would be just like other existing entities of boobrainz with some slight changes.

My First priority before GsoC starts would be getting familiar with other entity implementations.

During this phase I will invest my time understanding the codebase more intensly . And Also in the meantime I will work on the existing tickets to make BookBrainz more excellent.

Community Bonding Period :

Fix Existing bugs, help to merge pending PRs, and close issues.
Discuss with mentor about roadmap, Finalizing Database Schema and other the plan of action

Week 1-2 :

I will begin with setting up database (Creating new table) and Implementing the corresponding orm models and functions with tests

Week 3-4-5 :

I will begin with creating the web server routes and add the new entity saving saving mechanism.

Week 6-7:

Writing corresponding tests and cleaning up the code. Take reviews from mentor and make relevant changes. Units Test will be written using mocha and chai assertion library as already used for other entities in BookBrainz

Week 8 :

I Will begin Creating front-end entity create/edit/merge components (based on existing components for other entities)

Week 9 :

Catch up if the any frontend component/page to be created is lagging behind.

Update the elasticsearch indexing to make the series entity appear in the search results.(This might require a bit of help from mentor side)

Week 10 :

Clean up the code and write documentation. Discuss with the mentor relevant changes before the final submission of the work.

Stretch Goal

  • The addition of series endpoints in the existing API.
  • Add achievements for creating series.

Detailed information about yourself

My name is Akash Gupta( I go by @akashgp09 online), I am currently in the 2nd year of my Bachelor in Technology Degree from Kalinga Institute of Industrial Technology in Information Technology. I’ve always been fascinated by computers and the logic that made them tick. This was the major factor that resulted in me picking an interest in programming and deciding to follow the software development career path. I love working on web apps, as well as the related tools and technologies which make web apps possible and I have spent many of my nights up hacking on such projects.

I started contributing to BookBrainz from the 2nd week of March 2021.

Contributions in BookBrainz:

My PRs: Check out

My Commits: Check Out

Tell us about the computer(s) you have available for working on your SoC project!

I own a HP Probook X360 440G1, Core i5 8th Gen Processor with 16 GB RAM and 512 GB SSD running on Ubuntu 20.04.

When did you first start programming?

I started writing code in C++ in my 8th standard of school. I picked up some Python at secondary school and Javascript, C programming in my freshman year.

What type of music do you listen to? (Please list a series of MBIDs as examples.)

The choice of music mostly depends on the mood. My all time favorite music album is Starboy by The Weeknd aka Abel Makkonen Tesfaye.

If applying for a BookBrainz project: what type of books do you read?

I love to read. Some of my favourite books are Sherlock, Game of Thrones And Sacred Games.

What aspects of the project you’re applying for (e.g., MusicBrainz, AcousticBrainz, etc.) interest you the most?

As I mentioned, I love reading. Even when I learn something new, I always prefer reading the documentation instead of watching some video tutorials.

Not just reading but also I love writing. You can checkout out some of my blogs on Medium

Have you contributed to other Open Source projects? If so, which projects and can we see some of your code?
Yes, I have been involved in open source from the past 1 year.

Contributing to the development of Elastic Since January 2021, My Contributions include fixing bugs, adding features and writing tests.

I have made 19 PRs till now in Elastic with alone 16 Prs in The Elastic UI Framework

I have worked with Typescript, Jest , React, a11y Testing

My PRs : Check Out

How much time do you have available, and how would you plan to use it?

I would be able to devote approx 50 hours every week to gsoc. My classes will probably start in last 15 days. Work load will be less and I will be able to give 4-5 hours a day easily.

Do you plan to have a job or study during the summer in conjunction with Summer of Code?

No, I will be devoting all my time to GSoC

Last but not least:

I believe that I will finish the job at the end of happiness. Let’s enjoy it

2 Likes

Hi @mr_monkey, this is the initial draft for my proposal. Please have a look whenver you are free. I would love to hear some suggestions to improve this proposal.

1 Like

Solid proposal so far, nicely done :slight_smile:

Some comments:

I think we could do without the series_type table, we can probably use the existing entity_type enum

I’m not sure you need getSeriesRelationshipTargetByTypeId. If you know the series’ entity type and you know the "series contains $entity_type” relationship type id, you can just call getRelationshipTargetByTypeId with that id.

What the new utility function would do is however is the sorting.

I can think of different automatic sorting possibilities that could be useful, for example by alphabetical order. It would probably need passing an attribute to that sorting function (for example “position” or “defaultAlias.name”) and sort on that.

Some questions left unanswered:

  • When would we want alphabetical sorting over sorting by order of addition?
  • Would alphabetical sorting just be another ordering type? Or applied to certain entity types?
  • What happens if I edit a series with entities in it from auto to manual ordering type, considering the relationships will have null position? I guess an indication of when an entity was added to the series would be the relationship id, but I’m not sure that’s reliable.

Instead of the position attribute on series, I’d like to see a more generic relationship attributes system that could be used for other cases.
For example, that position would only be relevant to "series contains X” relationships, while for example we will want a way to describe that a relationship started and ended at a certain date (e.g. "Author_A was married to Author_B from start_date to end_date”).

I’d like to see you come up with some system that allows that flexibility, I think such a system will be very useful.

What I can think of at the top of my head would be:

  • tables for extra relationship attributes (one table per type e.g. relationship_order, relationship_date, etc.)
  • a column in the relationship table to point to an attribute row
  • a way to know which relationship attribute table to fetch from depending on the relationship type (if relationship type = “entity belongs to series”, then fetch rel. attribute from relationship_order table)

Happy to discuss all this in more detail if you want!

2 Likes

Yes Indeed, i see we can easily use the bookbrainz.entity_type enum as already used in other tables to restrict the types without creating a new table series_type for it :slight_smile:

Yes I agree, we can use our existing getRelationshipTargetByTypeId without creating a new one for series entity. However we will need a new utility function which will be handle the sorting stuff of a series entity.

This is great. I haven’t thought earlier for sorting with attribute other than position, but using other attributes like defaultAlias.name for sorting would really be great.

Not sure, but we can simply use this when we want to display the entities in the series in a arranged alphabetical order. Like suppose displaying series entity of type author in ascending order of their name.

Considering alphabetical sorting is only limited to entity name we can add another option alphabetical sorting to ordering_type. This will sort the entities in the series alphabetically according to the entity name. But if we want to sort entities with different attributes other than their name like for work series sorting work with work type(novel,non-fiction,short story, Epic, Anthology) than we will need a more complex setup (But i don’t think we need that and it wouldn’t be useful as well)

when we change the ordering type from automatic to manual then we will have to assign position to each entity before saving the edits.

Yes indeed, Instead of the position attribute on series, we can have a more generic relationship attributes system that could be used for other cases.

we can have the changes as suggested :

CREATE TABLE bookbrainz.relationship_order (
   id SERIAL PRIMARY KEY,
   position INT
);
CREATE TABLE bookbrainz.relationship_date (
   id SERIAL PRIMARY KEY,
   begin_year SMALLINT,
   begin_month SMALLINT,
   begin_day SMALLINT,
   end_year SMALLINT,
   end_month SMALLINT,
   end_day SMALLINT,
   ended BOOLEAN NOT NULL DEFAULT FALSE,
   CHECK (
		(
			(
				end_year IS NOT NULL OR
				end_month IS NOT NULL OR
				end_day IS NOT NULL
			) AND ended = TRUE
		) OR (
			(
				end_year IS NULL AND
				end_month IS NULL AND
				end_day IS NULL
			)
		)
	)
);
ALTER TABLE bookbrainz.relationship_order ADD FOREIGN KEY (id) REFERENCES bookbrainz.relationship (id);

ALTER TABLE bookbrainz.relationship_date ADD FOREIGN KEY (id) REFERENCES bookbrainz.relationship (id);

And in the frontend we will have date inputs which will describe when a relationship started and ended at a certain date.

I’m a bit skeptical about my approach and I know this will involve some more complex setup with more schema changes. I am eager to carry forward this discussion more in detail : )