Monday, April 30, 2007

A Python Plugin System

A couple of months ago André mentioned that having a plugin system in Crunchy would be "nice thing". I was feeling bored when I read his email, and so decided to go away and write one.

This post documents my attempt, which I like to think was actually rather successful.

Basically, Crunchy plugins are .py files that reside in a specific directory. They are automatically imported and initialised by the Crunchy core system on startup.

So the first thing we need to do is figure out what that special path is and enumerate all the .py files in it:

import os
import os.path
import imp

pluginpath = os.path.join(os.path.dirname(imp.find_module("pluginloader")[1]), "plugins/")
pluginfiles = [fname[:-3] for fname in os.listdir(pluginpath) if fname.endswith(".py")]

I'm afraid both lines are rather complicated, but they certainly demonstrate the power of Python. Because this code is run from within the module pluginloader we look up its (absolute) path and join that to the (relative) subpath plugins/. This gives us the absolute path to the plugin directory in a system-independent way.

The second line is (as you probably realised) a list comprehension, it first generates a list of the files in the plugin directory, then adds filters that list for files ending in .py, and finally removes the trailing .pys from the filenames to give a list of module names. All in one line!

Now that we have a list of modules, all we have to do is import them, which we can do with another list comprehension:

import sys

if not pluginpath in sys.path:
imported_modules = [__import__(fname) for fname in pluginfiles]

Once again, the Python code is disarmingly simple: all it does is add the plugin directory to the module import path and import all the modules, generating a list of the imported modulle objects along the way.

In the Crunchy Plugin API we ask that all initialisation code be placed in a custom register() function inside each plugin module. We can now call these functions with just one more line:

[mod.register() for mod in imported_modules]

And there you have it, a simple but powerful plugin system written in just 10 lines of Python code. Enjoy!

Edit: setuptools does the same job, with many more features and more flexibility.


Rob De Almeida said...

Just curious, have you looked at setuptools' entry points? Using entry points your plugins could live anywhere on the file system, and be distributed and installed separately from your main app.

Johannes Woolard said...

I had looked at them - but at the time they didn't quite seem to meet our requirements. In crunchy we actually have a complex system of dependencies between different plugins - so we have an intermediate stage between importing and registering that sorts plugins by their dependencies. My next post will probably be on this sorting.

PJE said...

If the sorting stage is "between importing and registering", you should be all set with setuptools, since it doesn't do anything past importing the specified objects. If the objects expose dependency information, you're good to go with your sorting.

Meanwhile, entry points handle such things as inter-package dependencies, and don't require modules to all be in a particular package or directory. Plugins can be installed as .egg files dropped into a "plugins" directory, then activated with find_plugins().

Johannes Woolard said...

Another reason for not using setuptools is our insistence on minimal dependencies; we aim to have only one external dependency for Crunchy: Python itself (at present we also require Firefox 1.5+).

I have to admit that setuptools are certainly worth a look - and seem to do the job extremely well.

Tcll said...

I cant seem to get your code to work entirely
"ImportError: No module named pluginloader"

I use python 2.5.4 (fav)
do I need to up my version of python,
or would this be safe in mine

^I like the dev of a 2.5 app because it should work in 2.6 and 2.7

anyways, a 2nd thought I had was maybe I need to create that file and place it in the app directory

but what goes in it??

Tcll said...
This comment has been removed by the author.
Tcll said...

you need the module '' in the app directory.

and it works with py25