GSoC 2018: Picard Plugin API v2


#1

Personal information

Nickname: Yagyansh
IRC nick: yagyanshbhatia
Email: yagyansh1999@gmail.com
GitHub: yagyanshbhatia
Time zone: UTC +5:30

Proposal


Overview

This proposal aims to replace the current Plugin API system with an improvised new system for Picard v2 , Provide development tools, Documentation and guidelines to plugin developers, and enforce i18n support.

Current system

The problems with the current plugin system are:

  • Over complicated code
  • No proper i18n support
  • No proper guidelines/developer-docs/Tools for plugin devs
  • Lot of time taken before publishing (Acceptance by community)
  • Dependency on MeB people to change/Upgrade/Maintain their code.
  • Increased workload on MeB people.

The New API

This proposal aims to Implement a new plugin API to overcome all above shortcomings. The code will be based on v2.0 release.


The key features of the new system will be:

  • i18n support
  • A Generic PicardPlugin class, where all hooks live in, which each plugin has to inherit.
  • Plugin author can fully test his plugin, without depending on PW or our team.
  • Publishing is easy, dev may or maynot register on PW, all he needs is link to the source.
  • No dependency on us to make changes/ upgrade plugin.
  • Better guidelines and Tools for developer.

Deliverables

With this proposal I’d like to set the following deliverables for my project:

  • Change Picard-Plugin-Tools to include:
    • Able to create Proposed directory structure and starter code.
    • Zip packaging
    • Managing manifest file
    • Parsing code and Generating .pot files, .mo files from the respective .po files
    • I18n support for plugins
    • Compatibility test for the final product
  • PW to have these functionalities:
    • Platform to Register a plugin
    • Maintain a sources list (Link to the manifest)
    • Display metadata from the manifest
    • Validity checks ( for the manifest, link for the .zip of plugin)
  • Picard tagger to have proposed new plugin API:
    • Able to install a plugin from link to the manifest (after verifying the crc32 checksum)
    • Able to install plugin from .zip file of the plugin
    • Retrieve the list of plugins from PW and Able to install them.
    • A generic superclass for all plugins that they all inherit ( PicardPlugin class )
    • Add sources without publishing, just with the link to manifest.
    • Display internationalised code (or default language if no translations provided).

Project plan and Implementation Details

Whole project can be seen as following sections:

  • New plugin structure, i18n handling
  • PicardPlugin class
  • Providing developer tools and documentation
  • Picard : Updating the code to handle and use to new plugin architecture.
  • Picard-Website : Updating the code to handle and use to new plugin architecture.

New Plugin Structure and i18n

    
  pluginA/
  |- manifest.json
  |- src/
      |- __init__.py
      |- something.py
  |- locale/
      |-en/
          |-LC_MESSAGES/
              |- en.po
              |- en.mo  
      |-fr/
          |-LC_MESSAGES/
              |- fr.po
              |- fr.mo 
  |- pluginA.pot
    
  

Plugin directory will have the above structure.
  A basic sample how the plugin will look and the steps to be followed is here.
  Instead of plugin source code itself, the manifest file will have Plugin’s metadata. Manifest file will look like this:

    
{
  "PLUGIN_NAME": "Video tools",
  "PLUGIN_AUTHOR": "Philipp Wolfer",
  "PLUGIN_DESCRIPTION": "Improves the video support in Picard by adding support for Matroska, WebM, AVI, QuickTime and MPEG files (renaming and fingerprinting only, no tagging) and providing $is_audio() and $is_video() scripting functions.",
  "PLUGIN_VERSION": "0.1",
  "PLUGIN_API_VERSIONS": [
    "\"1.3.0\""
  ],
  "PLUGIN_LICENSE": "GPL-2.0",
  "PLUGIN_LICENSE_URL": "https://www.gnu.org/licenses/gpl-2.0.html",
  "PLUGIN_ZIP_URL": "https://github.com/yagyanshbhatia/Code_snippets/archive/master.zip",
  "PLUGIN_ZIP_CRC32_CHECKSUM": "418ae8ec"
}
    
  

During registration, the author will provide the link to this manifest file, and PW will extract all these metadata and show on it. When Picard retrieves the source List from PW, it will use this manifest file and using the PLUGIN_ZIP_URL install the plugin.
  Each plugin author has to take care of his own translations. Each plugin directory will have its own .po , .pot and .mo files. They have to override _() and the N_() methods.
   Tools will archive the package and write its crc32 checksum in the manifest. This will be used to check the validity of the downloaded package on the user’s end.

PicardPlugin Class

Since plugins are an example of loose coupling, there needs to be a neutral location, somewhere between the plugins and the code that uses them, that each side of the system can look at, without having to know the details of the other side. This will be accomplished by defining a base class from which every plugin has to inherit.

  
class PicardPluginRegistry(type):
	plugins = []
	def __init__(cls, name, bases, atrrs):
		if name!='PicardPlugin':
			PicardPluginRegistry.plugins.append(cls)

Since the above class subclasses type, it can be used as a metaclass.



class PicardPlugin(object):
	__metaclass__ = PicardPluginRegistry
	"""
	mount point for plugins which refer to the action that can be performed
	==================================
	documentation, guidelines
	how to's and 
	required attributes here. 
      This is where we tell the developer how we expect the plugin to behave
	===================================
	"""
    def __init__(self, name):
        #initialise
        self.name = name
        self.enable = False

    def activate(self, trick):
        #enable the plugin
        self.enable = True

    def deactivate(self, trick):
        #disable the plugin
        self.enable = False

    def register_script_function(self, *args, **kwargs):
    	#code at https://github.com/yagyanshbhatia/picard/blob/12ed9ce63ae3a7cd5ae277cd4933c78c61430921/picard/script.py#L295

    def register_format(self, *args, **kwargs):
    	#code at https://github.com/yagyanshbhatia/picard/blob/23ea30b1d546372e81ccedeb63e3aade60004f31/picard/formats/__init__.py#L27
	"""
	all other hooks available in Picard are here. 
	====================
	Plugins don't need to know about the system,
	system doesn't need to know about how plugins are working. Duck typing at its finest.
	"""
    
  

This will serve as the mounting point for the new plugin architecture.
   A plugin is a class that inherits from PicardPlugin. A metaclass trickery makes sure that by the very act of inheriting from it, the plugin registers itself in the system.

This class is not meant to be instantiated, but serve as base classes for other classes to inherit from. This class will group methods and instance variables that will be used by a number of different subclasses into a common definition.
   The PicardPlugin class is incomplete by itself, but contains useful code that reduces the implementation burden of its subclasses.


def discover_plugins(path):
    """ 
    Iterate and Discover the plugin classes contained in Python files, given a path
    to scan. Return a list of plugin classes.
    path may point to a directory or a file ( can be a zip also)
    """
        mod = imp.load_module(modname, file, path, descr)
    return PicardPluginRegistry.plugins

This function can be used by the applications to find and load plugins. It gets a list of path in which to look for Python modules. Each module is loaded, which executes the class definitions within it. Those classes that inherit from PicardPlugin and get registered with PicardPluginRegistry, which can then be queried.

Fundamental aspects of the any plugin architecture are categorised as follows:

  1. Discovery
    the discover_plugins function implements this - plugins are Python classes that inherit from a known base class, contained in modules located in known places.
  2. Registration
    The very act of inheriting from the base class will register the plugin into the system, due to the peculiar construction of the metaclass.
  3. Defining a mount point for plugins to attach
    In our case, we have only one mount point, the PicardPlugin class.
  4. Extension points (expose the plugin functionalities back to Application)
    This is accomplished through “hooks” which can extend the functionalities provided by plugins to the application. The class will have all the available hooks in Picard

This is a fine example of the Duck architecture. The plugin framework itself has absolutely no expectation for how Plugin class is implemented, allowing maximum flexibility, and easy implementation.

Picard-Plugin-Tools and Documentation

This will involve Implementing code to produce the above mentioned directory structure, an interactive console to produce/Update the manifest file, generate the .pot file. From the .po files, tools will produce a .mo file.
  Documentation will help developers follow the guidelines, zip and publish their package, and test the package too.
  Picard-plugin-tools will use pygettext to help the author to parse the files and generate .pot file from which all .po files have to be made.
  Now, the catalog is built from .po files by msgformat.py which parses the .po files and generates the equivalent .mo files. See this for details and all the steps.
  Sample plugin and steps used to generate above files are here.
The crc32 hash of the zip archive will be calculated by the following script:


import zlib
def crc(fileName):
    prev = 0
    for eachLine in open(fileName,"rb"):
        prev = zlib.crc32(eachLine, prev)
    return "%X"%(prev & 0xFFFFFFFF)

Picard Website

Picard website will have a section where authors will Publish their plugins by just providing a link to the manifest file, in a very simple form like this.
  PW will retrieve the JSON data ( sample ) from the manifest file and check if the file is in the required proposed format, and whether the PLUGIN_ZIP_URL is valid. If it passes these tests, the plugin will be shown in the picard plugins page and will be included in the sources list, for main picard tagger to access.
  PW will only store the list for “sources” , i.e. the link to respective manifests.

Picard

Picard will install plugins through the following three ways:

  1. Through Picard website:
      Picard will have to retrieve the sources list from the PW, retrieve link to its manifest, hence providing all the metadata.
    From the PLUGIN_ZIP_URL , it will download the plugin package , validate the crc32 checksum and install it.

  2. Link to the manifest:
      User can directly provide the link to the desired manifest and hence no need to register on PW. This way developer would no longer be dependent on MeB circle and PW to publish the plugins.

  3. Local .zip package:
    User directly provides the .zip to the plugin, Picard extracts the package and reads the manifest file for metadata and install the extracted plugin

Security Considerations

Extensibility of any sort is cause for concern when it comes to security. Because plug-ins run their own code in the host application’s address space, there are essentially no theoretical limits on what they do with user’s application’s address space.
  We can warn the users about installing plug-ins from third parties and the side effects they may cause. What software is installed is really up to them, so we make sure they know the implications.

Possible extensions , To-do’s after GSoC

  • Migrate the existing plugins in picard-plugin to proposed new structure and host them individually, or ask the author to do so
  • Currently the current code coverage of picard sits at 28% which is way below our standard (60%), hence unittests need to be written for better code coverage.
  • A detailed plugin contribution documentation helping plugin developers to contribute
  • Port PW to py3
  • Port imp to importlb, as it is deprecated python 3.4 onwards

Proposed Project Timeline


Before 20 April

  1. Familiarise myself completely with current plugin system and its working.
  2. Set up all environments needed to save time in the actual coding phase.
  3. Research properly what and how I have to code, to compensate for the community bonding period.

6 May to 10 May

Get back in touch and prepare to start coding.

11 May to 25 May

Implement PicardPlugin class.

26 May to 31 May

Buffer

1 June to 15 June

Update Picard Website for the new system.


16 June to 20 June

Buffer

21 June to 10 July

Update main Picard tagger for the new system.

11 July to 22 July

Update Picard-Plugin-tools

23 July to 31 July

rigorous testing and bug fixes

31 July to 9 August

Buffer ( unexpected delays )

Detailed information about yourself

  • Tell us about the computer(s) you have available for working on your SoC project!

I have an HP-au118tx . On top of it, I have Ubuntu 16.04 . I shifted recently from Sublime to PyCharm and still exploring perks of using it as an IDE.

  • When did you first start programming?

I got introduced to code 4 years back (HTML/CSS) and started C programming 2 years ago. I dived into Python 6 months ago and since then I am fascinated by its easy and readable syntax, tons of awesome libraries, vibrant community and vast applications.

  • What type of music do you listen to? (Please list a series of MBIDs as examples.)

I’ve had phases, but these are my all time favorites:
Eminem - Not afraid (26b50a2d-d1fd-42ca-8370-4b71cd5ff8a5) , Rap god (26b50a2d-d1fd-42ca-8370-4b71cd5ff8a5). These days I am looping through shawn mendes - There’s nothing holding me back (94af5418-2180-41a6-ac3e-ec10bd66bd3d)

  • What aspects of the project you’re applying for (e.g., MusicBrainz, AcousticBrainz, etc.) interest you the most?

Being fascinated by python, I was exploring the possibilities with it. One fine day, I was exploring applications built on python and I came across Picard. I was always a music lover and now I discover an application based on python that handles music? That too open-source? I knew I had to go ahead then :wink:

  • Have you ever used MusicBrainz to tag your files?

Yes I’ve been using MusicBrainz Picard for almost three months now.

  • Have you contributed to other Open Source projects? If so, which projects and can we see some of your code?
    • If you have not contributed to open source projects, do you have other code we can look at?

No, nothing significant. checkout my GH for some of my projects. I did fix a couple of issues on JIRA so far.

  • What sorts of programming projects have you done on your own time?

You can checkout my GH for my projects. Major ones are Expense_Bot , Intelligent_faculty_homepage, and Search engine all hosted on my GH :slight_smile:

  • How much time do you have available, and how would you plan to use it?

I have my End semester examinations from 28 april to 5 may, hence a major part of community bonding period will be spent on that. I plan to compensate for that by starting early, solving bugs to get familiar with picard code more. I’ll also research more on what and how i have to code, so that i can make up for the time spent.

  • Do you plan to have a job or study during the summer in conjunction with Summer of Code?

None, if selected for GSoC.


#2

The first draft of my Proposal is complete. Any suggestions, improvisations and criticisms are welcome! :slight_smile: