Note: This idea is an extension of the idea here and discussion on IRC here.
Personal information
Name: Vishal Choudhary
IRC Nick: vishichoudhary1
Email: vishal129.vk@gmail.com
Github: vishichoudhary
Portfolio. vishi choudhary
Time Zone. UTC+05:30
Proposal
Present Scenario
Picard is apple of everyone’s eye for music tagging. Picard has some great functionalities which can further be enhanced by plugins. Presently, Picard supports multiple kinds of plugins.
Picard has a development cycle for plugins, which can be summarized as follow:
- Developer developed a plugin.
- To share with everyone, the developer needs to add it to PW plugins list. Which can be done by adding a plugin to GitHub repo.
- Website fetches sources of the plugins from GitHub repo and then generate zip files.
- The website then needs to restart every time in order to fetch the latest plugin from the repo.
- Only then, everyone can use that plugin.
Following diagram illustrates this cycle :
Issues with existing system:
Following are the major issues or missing things which we can work upon:
- The developer has to rely on Picard Maintenance team in order to add a plugin to PW.
- There is not any i18n support for plugins. Since Picard support i18n and plugins can manipulate Picard GUI too, So it's kind of missing feature in the existing system.
- There is not any centralized source for plugins hooks. Plugins need to import specific Picard file to work on.
- There is not well-defined plugin structure. Currently Picard’s plugin can be .py or .zip extension with their own structure, which causes problems like:
- Complex code to handle plugins.
- Extra complexity to fetch details of plugins, currently manifest are extracted from plugin code.
Proposal Overview:
My proposal includes:
- Creating a plugin framework for Picard, which includes following things:
- Creating a generic class, where all the available hooks are present and each plugin needs to inherit it.
- Add i18n support for plugins.
- Define basic plugin directory structure.
- Write code for registration of plugins and for accessing registered plugins.
- Create a new system for plugin development. Which have following points:
- Add a registration page for adding plugins in PW.
- Development of new API’s for proposed plugin system.
This allows the users to add the plugins from anywhere in Picard. So a developer need not depend upon Picard-developer community to distribute his/her plugin. A developer can simply share the link of a plugin to the users.
Deliverables
With this project, What I propose to do can be described in following subtasks:
1. Picard:
Change the Picard existing plugin handler code to adapt the new plugin structure. It include support i18n for plugins, a base class which plugins need to inherit.
2. Picard-Website:
Update Picard Website to hold plugins link. It will includes removing existing code to make zip files and an addition of some code to provide details of plugins and their locations.
3. Picard-Plugin-tool:
Refurbish Picard-Plugin-tool for new plugin structure. It comprises of creating the directory structure, managing manifest file, generating .pot file and packaging plugin as a zip file.
4. Plugins:
Port the existing plugins to fit for new plugin structure.
Implementation details and code organization
Structure of Plugins:
Currently Picard support .zip and .py extension plugin, without having much enforcement on plugin’s code structure. This make Picard’s plugin manager code a lot complex. So, having a defined plugin structure will ease Picard job to handle plugins.
Below is the proposed directory structure for Picard’s plugins:
PluginA/
├── manifest.json
├── po
│ ├── fr.po
│ └── PluginA.pot
└── src
└── __init__.py
- src/ directory will contain all the source code files of a plugin.
- po/ directory will have all the translation files of a plugin.
- Name of the plugin is similar to the name of the parent directory PluginA (in this case)
- Currently, manifest for plugins are extracted from the source code of plugins. This adds a lot more complexity in the code.
- Providing a manifest file will remove this extra complexity and it will be easy to get the information of plugins.
- The proposed manifest file of a plugin is shown below:
{
"PLUGIN_NAME": "Padded disc and tracknumbers",
"PLUGIN_AUTHOR": "Wieland Hoffmann",
"PLUGIN_VERSION": "1.0",
"PLUGIN_API_VERSIONS": [
"0.15.0",
"0.15.1",
"0.16.0",
"1.0.0",
"1.1.0",
"1.2.0",
"1.3.0",
"2.0"
],
"PLUGIN_LICENSE": "GPL-2.0 or later",
"PLUGIN_LICENSE_URL": "https://www.gnu.org/licenses/gpl-2.0.html",
"PLUGIN_DESCRIPTION": "Adds padded disc",
"PLUGIN_FILES": [
"__init__.py"
],
"PLUGIN_URL" : "https://www.github.com/vishichoudhary/snipets/sample_ports",
"PLUGIN_CHECKSUM" : "0e884a8d3c4f7f69a0089bce8638b459"
}
Picard’s Plugin Framework:
Aim of this architecture is to provide a simple but flexible plugin system for Picard.
While implementing plugin architecture, there are three things that are really necessary:
1) A way to declare a mount point for plugins.
There should exist some points, which plugins and Program can use. Without having to know the details of opposite side. Points are famously knowns as “Extension points”.
2) A way to register a plugin.
Program internal code need not to look around to find plugins that might work for it, there needs to be a way for plugins to announce their presence. Program only needs to care about the mount point
3) A way to retrieve the plugins that have been registered.
Once the plugins have done their thing at the mount point, the rest of the system needs to be able to iterate over the installed plugins and use them according to its need.
Mount point for plugins:
Mount Point is a place where extra features can be added. So a Mount point should be simple and flexible enough. Moving to a basic class for all plugins will be a great move.
- It will provide some few generic functions, each plugin needs to inherit it. And implement few methods like activate() and deactivate().
- All the available hooks will be provided by this class. So plugins need not call them directly.
- It will also serve as a documentation of all the hooks available in Picard.
Below is the rough idea of PluginBasic class:
import abc
from picard.plugin import PluginFunctions, PluginPriority
class PluginBasic(metaclass=abc.ABCMeta):
@abc.abstractmethod
def activate(self):
"""
Will activate the plugin, this function will be trigrred by enabling the plugin.
Every plugin need to inherit this.
"""
@abc.abstractmethod
def deactivate(self):
"""
Will deactivate the plugin, and will be triggered by disabling the plugin.
Every plugin need to inherit this.
"""
"""
Hooks available in the picard will be listed here.
"""
def register_track_metadata(*args, **kwargs):
track_metadata_processor.register(*args, **kwargs)
def unregister_track_metadata(*args, **kwargs):
track_metadata_processor.unregister(*args, **kwargs)
def register_album_metadata(*args, **kwargs):
album_metadata_processor.register(*args, **kwargs)
def unregister_album_metadata(*args, **kwargs):
album_metadata_processor.unregister(*args, **kwargs)
album_metadata_processor = PluginFunctions()
track_metadata_processor = PluginFunctions()
#all other processors
Registering a plugin:
Below is the rough code of updated PluginManager class which will walk in each plugin modules and import them using importlib module, put them in a dictionary. Plugin modules and manifest will be wrapped in a PluginWrapper class, which will them inserted in a dictionary with the key as plugin name.
class PluginManager:
def __init__(self):
self.plugins = {}
def load_plugindir(self, plugindir):
"""
1) check if plugin dir exist.
2) handle updates
3) call load_plugin for each plugin.
"""
def load_plugin(self, name, plugindir):
path = os.path.join(plugindir, name)
manifest = manifest_from_json(os.path.join(path, "manifest.json"))
plugin_modules = {}
for plugin in os.listdir(os.path.join(path, "src")):
"""
plugin_modules[plugin] = module
"""
plugin = PluginWrapper(plugin_modules, manifest)
self.plugins[name] = plugins
Activate/Deactivate Plugins:
When we have to use registered plugins, we need to call their entry point. Since we have all the parts of plugins stored somewhere in Picard. We can simply obtain the obj of plugin from the PluginManager plugins dictionary. And invoke the plugin when needed. Following code gives a very basic illustration of same.
manager = PluginManager()
manager.load_plugindir(PLUGIN_DIR)
plugin = manager.plugins[plugin_name]
plugin_obj = plugin.modules[name].Plugin()
plugin_obj.activate()
plugin_obj.deactivate()
I have created a small simulator which illustrates:
- How to load plugins
- How to run them
- How to register/unregister them
And can be found at Picard Simulator.
Installation of plugin and i18n:
Plugins will be downloaded from sources configured by the user.
A Plugin will be downloaded as a zip file but will be configured in plugin directory as of structure shown above.
For i18n, each plugin has to override its _() and N_() methods. Each plugin has to care about it’s own translation.
Picard will use its own domain and plugins will use their own.
To prevent naming conflicts we can use the names of plugins as their domains.
Option To Add Plugin Locally.
There will be two option to add a plugin locally.
- Add a remote location.
- Local archives.
Add a remote location:
User interface design is essential for numerous reasons. Firstly the more perceptive the UI the easier it is to use. The better the user interface the easier it is to guide people to use it.
Along with providing the facility to download plugins from anywhere, it’s our prime motive to provide the user a simple and straightforward interface. So users will be provided with an option in Picard to add, edit and remove the link.
Below is the screenshot illustrating the same :
Local archive files:
- Extracting the archives in appropriate folder.
- Validating the plugin.
- Registering the plugin.
Port from imp module to importlib
Picard uses python imp module to load plugins which have been deprecated from python3.4. So it will require to port from imp module to importlib module.
Presently used module and their replacement is following:
-
imp.find_modulewill be replaced byimportlib.util.find_specofficial docs -
imp.load_modulewill be replaced byimportlib.import_moduleofficial docs
Updation of Picard-Website:
Update Picard-Website such that it will hold the links of plugins and shows the information of
plugins. This includes:
- Make API for new plugin system.
- Providing a registration page for developers to add plugins.
- Provide documentation.
- Port from python2 to python3.
- Set jobs to validate archives against manifest files.
Information of plugins will be extracted from their manifest files.
Documentation of plugin development will also be provided in the Picard-Website. This will
include providing information about:
Metadata Processor
Context Menu Actions
File Formats
Tagger Script Functions
A sample plugin development tutorial will also be provided in the Picard-Website.
Inspiration source. OpenStack
For user convenience, documentation about every Picard tool should be made available in one place. So documentation about Picard-Plugin-tool will also be included in Picard-Website.
Plugin Api v3
As Picard 2.1 is going to have a new plugin structure. So, Picard v1 plugins and v2 plugins will be incompatible with it.
So new endpoints need to implement, Presently there are two endpoints
picard.musicbrainz.org/api/<version>/pluginspicard.musicbrainz.org/api/<version>/download?id=<name>
Since, now we are not hosting any plugins onto picard website. We don’t require download
endpoint for Api v3. We only need plugin endpoint.
picard.musicbrainz.org/api/v3/plugins
will be additional api with single end point. And Json data returned by api will be following:
Picard-Plugin-tool Refurbishment:
Providing the plugin developers with the tool to manage their plugins will help them and will attract
developers to develop more plugins. So, it is very important to provide a powerful tool. Picard-
Plugin-tool is providing the same functionalities but it needs to change a little bit. It includes
providing the following options:
1. Create Directory Structure:
It will create a defined directory structure for a plugin.
ppt mkdir <name of plugin>
2. Package as zip:
Plugins will be distributed as zip files. So this will help users to zip their plugin.
ppt zip <path of plugin>
3. Generate .pot file:
To provide i18n support. Plugins need to have thier own .pot files. This will generate .pot file
from .py files.
ppt genpot <path of src/ of plugin>
Port existing plugins:
Presently Picard has more than 30 plugins, which needs to be port in the new plugin structure.
Sample of ported plugin can be found on padded_disc_and_tracknumbers
How to register your plugin ?
Screenshot below illustrates how a developer can register his/her plugin.
Developers needs to login with musicbrainz id and have to provide the link of manifest , complete captcha and finally hit submit button.
Sign in
Register
Validation:
After submission PW can retrieve archive and validate checksum against manifest. If it doesn’t match, message will be shown to submitter.
Timeline:
Community Bonding (April 24 – May 13)
Spend this time with community to find out what exactly I have to code. Also, discuss design
decisions with the mentor.
Phase 1 (May 14 – June 11)
- Porting Picard existing code to install/update plugins.
- Apart from this, port the code from imp module to importlib module as imp module happens
to be deprecated in python3.official docs - Implement a base class, Which will define the basic interface and provide some functions for all plugins.
- Each plugin should inherit it and implement some methods. This class will also hold all available hooks. And will also serve as documentation.
Phase 2 (June 12 – July 9)
- I aim to do Picard-Website work in this phase. This will involve removing the existing code of creating zip files from Picard plugin repo. And addition of some code to make it hold plugins link and displaying plugins information.
- This phase would also involve the addition of some code in picard-plugin-tool to create directory structure, managing manifest, and packaging plugin.
Phase 3 (July 10 – August 6)
This phase would involve porting existing Picard plugins to new plugin-structure, clean up of the code written in earlier phases, bug fixes and add some new tests to make sure that everything works as it is expected to be.
After GSoC:
Continue working on Picard as I have been since the last September. I will contribute to release Picard 2.1. Optional idea I have in my mind is providing a option to add skins in Picard to change its UI. I haven’t done much research on this. Concurrency code enhancement of Picard seems challenging but would like to give it a try.
A detailed timeline of my work is as follows:
• Week 1: Begin with setting up the environment. Defining generic class.
• Week 2: Continue working on generic class.
• Week 3: Work on Plugin handler code ,and write tests.
• Week 4: Solve bugs if there’s any else prepare for Phase I evaluation. PHASE I Evaluation here
• Week 5: Taking mentor evaluation into account, fix stuff in code written so far and continue coding.
• Week 6: Begin with docker setup for Picard-Website
• Week 7: Update Picad-Website
• Week 8: Update Picard-plugin-tool and complete documentation. PHASE 2 evaluation here
• Week 9: Start porting plugins
• Week 10: Complete any pending stuff.
• Week 11: Test if the plugins are working the same as they were before porting.
• Week 12: Complete previous pending task, if any.Solve bugs
• Week 13: Buffer week. Work on final submission.
Detailed Information about yourself
I am a junior CS undergrad at the National Institute of Technology, Hamirpur. I came across Picard
when looking for a music tagger last August. As I was on Arch Linux, I got a Picard dev version and
found out a bug in the search bar. I fixed that issue and have been helping out in development since then.
Here is a list of pull requests I’ve worked on over time. I have a semi-active blog which i would like
to update regularly over the Summer of Code period about my progress with the project.
Question: Tell us about the computer(s) you have available for working on your SoC project?
Answer: Currently I have an Acer laptop with Amd A8 processor and 8 GB RAM, running Manjaro linux.
Question: When did you first start programming?
Answer: I have been programming since 10th grade, mostly html/css and was introduced to C
programming 5 years ago. I picked up Python in my freshman year, working on small applications.
Question: What type of music do you listen to?
Answer: I listen mostly pop and folk songs. My Favourite artists and songs are:
Adele, Ed sheeran, Gurdas Maan,
ColdPlay, Arijit Singh. Presently top listen are: Perfect, Something just like this.
Question: Have you ever used MusicBrainz to tag your files?
Answer: Yes I use Picard to tag my files.
Question: Have you contributed to other Open Source projects? If so, which projects and can we see some of your code?
Answer: I have contributed to the source code of Picard, and various other projects which can be
found on my GH.
Question: What sorts of programming projects have you done on your own time?
Answer: I have worked on Ball and Bridge game. And a video chat app. And various other college level projects. Please check my GH and my portfolio.
Question: How much time do you have available, and how would you plan to use it?
Answer: I have holidays during most of the coding period and can work full time (45-50 hrs per
week) on the project.
Question: Do you plan to have a job or study during the summer in conjunction with Summer of Code?
Answer: None, if selected for GSoC.