A Python Plugin System

A couple of months ago André mentioned that having a plugin system in Crunchy would be "nice thing". I was feeling bored when I read his email, and so decided to go away and write one.

This post documents my attempt, which I like to think was actually rather successful.

Basically, Crunchy plugins are .py files that reside in a specific directory. They are automatically imported and initialised by the Crunchy core system on startup.

So the first thing we need to do is figure out what that special path is and enumerate all the .py files in it:

import os
import os.path
import imp

pluginpath = os.path.join(os.path.dirname(imp.find_module("pluginloader")[1]), "plugins/")
pluginfiles = [fname[:-3] for fname in os.listdir(pluginpath) if fname.endswith(".py")]

I'm afraid both lines are rather complicated, but they certainly demonstrate the power of Python. Because this code is run from within the module pluginloader we look up its (absolute) path and join that to the (relative) subpath plugins/. This gives us the absolute path to the plugin directory in a system-independent way.

The second line is (as you probably realised) a list comprehension, it first generates a list of the files in the plugin directory, then adds filters that list for files ending in .py, and finally removes the trailing .pys from the filenames to give a list of module names. All in one line!

Now that we have a list of modules, all we have to do is import them, which we can do with another list comprehension:

import sys

if not pluginpath in sys.path:
    sys.path.insert(pluginpath)
imported_modules = [__import__(fname) for fname in pluginfiles]

Once again, the Python code is disarmingly simple: all it does is add the plugin directory to the module import path and import all the modules, generating a list of the imported modulle objects along the way.

In the Crunchy Plugin API we ask that all initialisation code be placed in a custom register() function inside each plugin module. We can now call these functions with just one more line:

[mod.register() for mod in imported_modules]

And there you have it, a simple but powerful plugin system written in just 10 lines of Python code. Enjoy!

Edit: setuptools does the same job, with many more features and more flexibility.

...and back again...

Hi, again.

I've been lazy, and let this lapse (I know, no posts since July last year!). Now I want to resurrect my blog - in order to post some interesting things that I've been working on.

Firstly some news about Crunchy: Since I last posted it has been renamed from Crunchy Frog to just plain Crunchy. This was because of another project called CrunchyFrog. The AJAX based IO system that I mentioned last July has finally come together in the last couple of months as part of a complete rewrite of Crunchy - with a plugin-based architecture (more on that soon), a very neat HTTP Server (with COMET capabilities) &c &c.

And finally some personal news: I successfully completed the Summer of Code last year as a student, and am acting as a mentor this year (for Bryan Psimas - a very promising student from the states). I'm also working in the City of London this year: as a techie at an investment bank.

Lights and Sound

Among quite a few small things that I've been working on over the last week, I've added `proof of concept` Sound Synthesizer and some fancier formatting to the output.

The sound synthesizer has no dependencies outside of the Python built-in library modules (wave, ossaudiodev and winsound) and should be completely cross-platform (OK, maybe not mac-compatible :-( ), although I haven't had a chance to test it on Windows yet. Admittedly this isn't really a feature of crunchy, more a demo of what is possible with crunchy - but it should still be interesting for kids.

The formatting uses HTML in the webpage to show stdout output blue and stderr output red, which makes errors that bit more readable. The project to handle errors more gracefully has stalled somewhat because of the huge number of different ways most errors can occur - for instance SystemError - you really can't do much with it other than display the whole lot.

One problem that has become more pressing with the introduction of the SoundSynth is the way program execution blocks the interface. I'm working on a way of simulating a remote tty using AJAX which will solve both the asynchronous execution and the raw_input problems. I don't have all that much experience at threaded programming, so it's very much a learning experience for me!

Once we have integrated the graphics canvas into the normal interpreter (so that graphics commands can be called from anywhere within crunchy and they will still load) I think we'll be ready for a new release of crunchy and I will be ready to start writing tutorials.

Say no to temporary files!

My last post mentioned the problems I was having with doctests, luckily I managed to solve them - but only after about 5 hours of frustrated playing with the API.

The problem arose because to get a DocTest object you have to know the namespace over which it is to be tested, but getting that namespace in the right place is tricky. My final solution was (I like to think) quite an inspired hack:

1. Take the code to generate the namespace and append some code to generate and run a doctest.
2. generate DocTestParser and DocTestRunner objects.
   
3. exec the code in a namespace that has references to the DocTestParser, the DocTestRunner and a string containing the actual doctests.

The end result looks something like this:

def run_doctest(code, test):
....code = code + '\n__dtest=__parser.get_doctest(__test, globals(), "Crunchy Doctest", "crunchy", 0)\n__runner.run(__dtest)\n'
....runner = DocTestRunner()
....parser = DocTestParser()
....exec code in {'__runner':runner, '__parser':parser, '__test':test}

It's a wonderful feeling when something is suddenly so obvious!

AJAX Joy

Over the last two days I've been playing with HTTP POST and the xmlhttprequest object and teaching myself the HTML DOM (basically the stuff underneath AJAX, although without the AX). Crunchy now only uses HTTP GET for loading new pages and the graphics canvas (which will change as soon as I've tidied up what I've done already)

The speed improvements feel incredible  - pages no longer even pause when running most simple code.

Another change I've made is stopping code executing in temporary files, sadly I can't see how to do doctests without writing to a temporary file because we need a module to run doctest in and as far as I know, module objects are only created when a file is read.

Another welcome side effect of these changes is that it will be almost trivial to add in support for custom error reporting, we just have to subclass code.InteractiveInterpreter - isn't Python beautiful?!

I'll be posting a request for comments on simplified error reporting on the edu-sig mailing list in a couple of days once I've come up with some decent ideas.

In other news, I'm starting to doubt the viability of teaching programming to 7 year old's without recourse to an adult on hand all the time - but on the other hand Crunchy is becoming a viable platform even for more advanced students (Andrew Harrington from Loyola University seems very interested in it). I will continue to endeavour to come up with ways of making this easy enough for kids <10 years old, but I feel it would be more constructive to concentrate on an age group of >10. A tutorial that a 10 year old can understand without adult guidance should be possible for a 7 year old with an adult!