tag:blogger.com,1999:blog-282362922024-03-12T23:14:17.790+00:00CrunchyA Pythonic Learning ExperienceJohannes Woolardhttp://www.blogger.com/profile/11444047692360756346noreply@blogger.comBlogger14125tag:blogger.com,1999:blog-28236292.post-25799908925448164552007-05-17T22:31:00.001+01:002007-05-18T18:29:03.143+01:00Python CGI HowtoI'm trying to use this blog as a place to document my new experiences with Python: some of them are very basic (like this post) and some are somewhat more advanced (for example the internal workings of <a href="http://code.google.com/p/crunchy">Crunchy</a>). So please don't be surprised at my "noobie" take on some things...<br /><br />Anyway, the other day I was playing around with <a href="http://hacketyhack.net">HacketyHack</a> (just to <span style="font-style:italic;">know my enemy</span>, as it were). Overall, I was quite impressed by it - although it does have slightly different aims to Crunchy. One of the best "features" of HacketyHack is its overall polish, and this was largely due to its excellent website. So I decided to make use of the Sourceforge webspace that Crunchy has had for a while (it used to be our main site) and at the same time learn how CGI works.<br /><br />In case you didn't know, CGI stands for <span style="font-style:italic;">Common Gateway Interface</span> and allows web pages to be generated on the fly by arbitrary programs. CGI works by literally sending printed output from the program down a pipe to the user's browser. Python, of course, has libraries to make CGI scripting easy. Here is a very simple python CGI script:<br /><pre title="py_code"><br />#!/usr/bin/python<br />import cgi<br /><br />cgi.test()<br /></pre><br />I know that normally you wouldn't worry about that first line, but for CGI scripts it is absolutely essential! (without it you get an <span style="font-style:italic;">Internal Server Error</span>). The second line imports Python's special CGI helper library and which includes a useful test function that we call in the final line. To make it work you have to place this script in your <span style="font-style:italic;">cgi-bin/</span> directory and give it global read and execute permissions using the *nix command:<br /><pre><br />$ chmod +rx <span style="font-style:italic;">filename</span><br /></pre><br />Once you've done this you can visit the page in your browser (it will probably be at a path something like <span style="font-style:italic;">/cgi-bin/test.py</span>, depending on how you named your script). You should see a detailed description of the environment in which you're script is running, but it won't look very pretty.<br /><br />Writing a proper hello world CGI script is very simple, it looks like this:<br /><pre title="py_code"><br />#!/usr/bin/python<br /><br />print "Content-type: text/html"<br />print<br /><br />print "<html><body>Hello World</body></html>"<br /></pre><br />Some more explanation: we don't actually need the cgi helper library here, so we don't load it. The first line of python code prints out a MIME-type that tells the browser what is coming next, then we print a blank line to indicate that we have finished generating headers, finally we print out the HTML code that we want to send.<br /><br />Of course this is just a very short introduction to CGI scripting with Python, and if you want to create a larger dynamic website in Python you're probably better off using one of the Python web frameworks (like <a href="http://twistedmatrix.com/trac/">Twisted</a> or <a href="http://pylonshq.com/">Pylons</a>). One day I might write a post about them...Johannes Woolardhttp://www.blogger.com/profile/11444047692360756346noreply@blogger.com0tag:blogger.com,1999:blog-28236292.post-37494150111177569442007-05-03T09:17:00.000+01:002007-05-04T16:41:13.892+01:00SVN Merging using SubClipseUp until a few days ago I was laboriously merging in changes in Subclipse by hand: copying files across manually, trying to make sure I didn't miss anything. Then I discovered the merge tool: Now I can merge in complex changes that would have taken hours in just a few minutes.<br /><br />The process is remarkably simple: Start by right click the root of the tree that you want the code to be merged into (in the file browser) and go to <span style="font-style:italic;">Team -> Merge</span>. This gives you a somewhat unintuitive dialog: it took me a while to figure out that it was only asking for a source to merge from (even though one section does say <span style="font-style:italic;">To:</span>). For most reasonable cases you want to use the same SVN path in the <span style="font-style:italic;">From:</span> and the <span style="font-style:italic;">To:</span> sections. In the <span style="font-style:italic;">From</span> revision box you will want to put the revision which you last merged in, and in the <span style="font-style:italic;">To</span> section select <span style="font-style:italic;">Head Revision</span>.<br /><br />Now you're ready to go: just hit <span style="font-style:italic;">Merge</span> and SVN will work its magic (although it might take a while - be patient). Once it has finished it will display a log of its actions in the Console window, most of it will have worked fine - but there will probably be some conflicts, marked with a red <span style="color: red; font-style:italic;">C</span>. All you have to do here is locate the relevant files in the file browser, right click on them and hit <span style="font-style:italic;">Team -> Edit Conflicts</span>. This will open a new editor window with two panes, you're version on the left and the other version on the right.<br /><br />In the Edit Conflicts window all conflicts will be highlighted, and all you have to do is resolve them by making suitable changes to the left-hand pane and saving (this is easy: they're rarely more than a few lines of code). Once things are sorted out, right click on the file in the file browser again and select <span style="font-style:italic;">Team -> Mark as Resolved</span>. Once you've resolved all the conflicts you're done. Subclipse really is a wonderful piece of software.<br /><br />One last tip: I find it useful to commit my branch just before merging changes: that way if something goes wrong I just have to hit <span style="font-style:italic;">Team -> Revert</span> and everything is better again.<br /><br /><span style="font-weight:bold;">Note:</span> For those of you who haven't tried Eclipse, <a href="http://subclipse.tigris.org/">Subclipse</a> is an <a href="http://www.eclipse.org/">Eclipse</a> plugin that adds SVN support - I use it in conjunction with <a href="http://pydev.sourceforge.net/">Pydev</a>.Johannes Woolardhttp://www.blogger.com/profile/11444047692360756346noreply@blogger.com11tag:blogger.com,1999:blog-28236292.post-1913195743723701972007-04-30T15:16:00.000+01:002007-05-01T02:01:19.745+01:00A Python Plugin SystemA couple of months ago <a href="http://aroberge.blogspot.com">André</a> mentioned that having a plugin system in Crunchy would be "nice thing". I was feeling bored when I read his email, and so decided to go away and write one.<br /><br />This post documents my attempt, which I like to think was actually rather successful.<br /><br />Basically, Crunchy plugins are .py files that reside in a specific directory. They are automatically imported and initialised by the Crunchy core system on startup.<br /><br />So the first thing we need to do is figure out what that special path is and enumerate all the .py files in it:<br /><br /><pre title="py_code"><br />import os<br />import os.path<br />import imp<br /><br />pluginpath = os.path.join(os.path.dirname(imp.find_module("pluginloader")[1]), "plugins/")<br />pluginfiles = [fname[:-3] for fname in os.listdir(pluginpath) if fname.endswith(".py")]<br /></pre><br /><br />I'm afraid both lines are rather complicated, but they certainly demonstrate the power of Python. Because this code is run from within the module <span style="font-style:italic;">pluginloader</span> we look up its (absolute) path and join that to the (relative) subpath <span style="font-style:italic;">plugins/</span>. This gives us the absolute path to the plugin directory in a system-independent way.<br /><br />The second line is (as you probably realised) a list comprehension, it first generates a list of the files in the plugin directory, then adds filters that list for files ending in <span style="font-style:italic;">.py</span>, and finally removes the trailing <span style="font-style:italic;">.py</span>s from the filenames to give a list of module names. All in one line!<br /><br />Now that we have a list of modules, all we have to do is import them, which we can do with another list comprehension:<br /><br /><pre title="py_code"><br />import sys<br /><br />if not pluginpath in sys.path:<br /> sys.path.insert(pluginpath)<br />imported_modules = [__import__(fname) for fname in pluginfiles]<br /></pre><br /><br />Once again, the Python code is disarmingly simple: all it does is add the plugin directory to the module import path and import all the modules, generating a list of the imported modulle objects along the way.<br /><br />In the Crunchy Plugin API we ask that all initialisation code be placed in a custom <span style="font-style:italic;">register()</span> function inside each plugin module. We can now call these functions with just one more line:<br /><br /><pre title="py_code"><br />[mod.register() for mod in imported_modules]<br /></pre><br /><br />And there you have it, a simple but powerful plugin system written in just 10 lines of Python code. Enjoy!<br /><br /><span style="font-weight:bold;">Edit:</span> <a href="http://cheeseshop.python.org/pypi/setuptools">setuptools</a> does the same job, with many more features and more flexibility.Johannes Woolardhttp://www.blogger.com/profile/11444047692360756346noreply@blogger.com8tag:blogger.com,1999:blog-28236292.post-27468052620413009412007-04-30T15:00:00.000+01:002007-04-30T14:54:02.631+01:00...and back again...Hi, again.<br /><br />I've been lazy, and let this lapse (I know, no posts since July last year!). Now I want to resurrect my blog - in order to post some interesting things that I've been working on.<br /><br />Firstly some news about Crunchy: Since I last posted it has been renamed from <span style="font-style: italic;">Crunchy Frog</span> to just plain <span style="font-style: italic;">Crunchy.<span style="font-style: italic;"> </span></span>This was because of another project called <a href="http://www2.blogger.com/img/gl.link.gif"><span style="font-style: italic;">CrunchyFrog</span></a>. The AJAX based IO system that I mentioned last July has finally come together in the last couple of months as part of a complete rewrite of Crunchy - with a plugin-based architecture (more on that soon), a very neat HTTP Server (with COMET capabilities) &c &c.<br /><br />And finally some personal news: I successfully completed the Summer of Code last year as a student, and am acting as a mentor this year (for Bryan Psimas - a very promising student from the states). I'm also working in the City of London this year: as a techie at an investment bank.Johannes Woolardhttp://www.blogger.com/profile/11444047692360756346noreply@blogger.com0tag:blogger.com,1999:blog-28236292.post-1152350568629157222006-07-08T09:56:00.000+01:002006-07-08T10:24:29.536+01:00Lights and SoundAmong quite a few small things that I've been working on over the last week, I've added `proof of concept` Sound Synthesizer and some fancier formatting to the output.<br /><br />The sound synthesizer has no dependencies outside of the Python built-in library modules (wave, ossaudiodev and winsound) and should be completely cross-platform (OK, maybe not mac-compatible :-( ), although I haven't had a chance to test it on Windows yet. Admittedly this isn't really a feature of crunchy, more a demo of what is possible with crunchy - but it should still be interesting for kids.<br /><br />The formatting uses HTML in the webpage to show stdout output blue and stderr output red, which makes errors that bit more readable. The project to handle errors more gracefully has stalled somewhat because of the huge number of different ways most errors can occur - for instance SystemError - you really can't do much with it other than display the whole lot.<br /><br />One problem that has become more pressing with the introduction of the SoundSynth is the way program execution blocks the interface. I'm working on a way of simulating a remote tty using AJAX which will solve both the asynchronous execution and the raw_input problems. I don't have all that much experience at threaded programming, so it's very much a learning experience for me!<br /><br />Once we have integrated the graphics canvas into the normal interpreter (so that graphics commands can be called from anywhere within crunchy and they will still load) I think we'll be ready for a new release of crunchy and I will be ready to start writing tutorials.Johannes Woolardhttp://www.blogger.com/profile/11444047692360756346noreply@blogger.com0tag:blogger.com,1999:blog-28236292.post-1151615100533155872006-06-29T22:01:00.000+01:002006-06-29T23:35:26.140+01:00Say no to temporary files!<p>My last post mentioned the problems I was having with doctests, luckily I managed to solve them - but only after about 5 hours of frustrated playing with the API.</p><p>The problem arose because to get a DocTest object you have to know the namespace over which it is to be tested, but getting that namespace in the right place is tricky. My final solution was (I like to think) quite an inspired hack:</p><ol><li>Take the code to generate the namespace and append some code to generate and run a doctest.</li><li>generate DocTestParser and DocTestRunner objects.<br /></li><li>exec the code in a namespace that has references to the DocTestParser, the DocTestRunner and a string containing the actual doctests.</li></ol><p>The end result looks something like this:</p><blockquote style="font-family: courier new;">def run_doctest(code, test):<br />....code = code + '\n__dtest=__parser.get_doctest(__test, globals(), "Crunchy Doctest", "crunchy", 0)\n__runner.run(__dtest)\n'<br />....runner = DocTestRunner()<br />....parser = DocTestParser()<br />....exec code in {'__runner':runner, '__parser':parser, '__test':test}</blockquote>It's a wonderful feeling when something is suddenly so obvious!Johannes Woolardhttp://www.blogger.com/profile/11444047692360756346noreply@blogger.com1tag:blogger.com,1999:blog-28236292.post-1151446066440257562006-06-27T23:04:00.000+01:002006-06-27T23:07:46.463+01:00AJAX Joy<div xmlns="http://www.w3.org/1999/xhtml"><p>Over the last two days I've been playing with HTTP POST and the xmlhttprequest object and teaching myself the HTML DOM (basically the stuff underneath AJAX, although without the AX). Crunchy now only uses HTTP GET for loading new pages and the graphics canvas (which will change as soon as I've tidied up what I've done already)</p> <p>The speed improvements feel incredible - pages no longer even pause when running most simple code. <br/> </p> <p>Another change I've made is stopping code executing in temporary files, sadly I can't see how to do doctests without writing to a temporary file because we need a module to run doctest in and as far as I know, module objects are only created when a file is read.</p> <p>Another welcome side effect of these changes is that it will be almost trivial to add in support for custom error reporting, we just have to subclass code.InteractiveInterpreter - isn't Python beautiful?!</p> <p>I'll be posting a request for comments on simplified error reporting on the edu-sig mailing list in a couple of days once I've come up with some decent ideas.</p> <p>In other news, I'm starting to doubt the viability of teaching programming to 7 year old's without recourse to an adult on hand all the time - but on the other hand Crunchy is becoming a viable platform even for more advanced students (Andrew Harrington from Loyola University seems very interested in it). I will continue to endeavour to come up with ways of making this easy enough for kids <10 years old, but I feel it would be more constructive to concentrate on an age group of >10. A tutorial that a 10 year old can understand without adult guidance should be possible for a 7 year old with an adult!<br/> </p> </div>Johannes Woolardhttp://www.blogger.com/profile/11444047692360756346noreply@blogger.com2tag:blogger.com,1999:blog-28236292.post-1151246488426016812006-06-25T15:34:00.000+01:002006-06-25T15:41:28.446+01:00Back!I know it's been 3 weeks since I last posted, I've been suffering from chronic exams and revision. Now I'm finally back home and working hard again.<br /><br />There has been lots of progress of crunchy - we have a Graphics Canvas (coded by Andre) using javascript and the canvas tag, Andre has squashed a whole load of bugs and added a proper error interface.<br /><br />Over the next few days I will be working on a system to parse errors in student's code and present the students with simplified/more meaningful error messages. I'll also start making POST requests work in preparation for moving a lot of the functionality over to AJAX.Johannes Woolardhttp://www.blogger.com/profile/11444047692360756346noreply@blogger.com0tag:blogger.com,1999:blog-28236292.post-1149110462215786772006-05-31T21:54:00.000+01:002006-05-31T22:21:02.240+01:00Progress......has been slow this week - I have mocks on Thursday and Friday.<br /><br />There have been lots of helpful suggestions for a graphics library varying from feature requests (The need to address individual pixels) to heavy duty discussions on the pedagogical aspects of graphics.<br /><br />I started coding a library at the beginning of the week, but I have since realised that it was far too complex for a 7 year old. The <a href="http://www.livewires.org.uk/python/">Livewires</a> API looks interesting but is aimed at 12+, maybe 7 year olds are a bit ambitious. Platform independence is another issue: Cairo is all well and good on UNIX, but it has stupid dependencies on Windows - there is a large part of me that wants to use Cairo on *NIX (it even has SVG support, imagine the possibilities!) and fall back on something less impressive for Windows. Or maybe I should create my own distribution of PyCairo for Windows that doesn't depend on GTK.<br /><br />I have also just realised that sound is going to be an issue - PyGame has excellent support for playing pregenerated sound, but virtually none for synthesized sound, which was what I had envisaged. Some kind of interface to MIDI would be perfect - although maybe too much to hope for. I don't want to have to rely on pre-recorded sound, it's much less interesting.<br /><br />Anyway, end of winge. There won't be another status update until Sunday at the earliest I'm afraid.Johannes Woolardhttp://www.blogger.com/profile/11444047692360756346noreply@blogger.com2tag:blogger.com,1999:blog-28236292.post-1148659114442035492006-05-26T16:56:00.000+01:002006-05-26T16:59:03.770+01:00Code ExplanationsI've created a page where I will put explanations of my code: <a href="http://crunchy.python-hosting.com/wiki/WalkThroughs">http://crunchy.python-hosting.com/wiki/WalkThroughs</a>.<br /><br />Think of the stuff here as more advanced tutorials that don't fit in with the 7-14 target age range!Johannes Woolardhttp://www.blogger.com/profile/11444047692360756346noreply@blogger.com0tag:blogger.com,1999:blog-28236292.post-1148652983770725532006-05-26T14:37:00.000+01:002006-05-26T15:16:23.786+01:00Small UpdateAndre found a solution to the exiting problem - I had just been setting the wrong variable!<br /><br />In other news, I have registered an account with <a href="http://python-hosting.com">python-hosting.com</a> which I will be using as an SVN repository - at the moment I don't have any plans to use the wiki apart from the front page: take a look at <a href="http://crunchy.python-hosting.com">crunchy.python-hosting.com</a>.Johannes Woolardhttp://www.blogger.com/profile/11444047692360756346noreply@blogger.com0tag:blogger.com,1999:blog-28236292.post-1148604179539002972006-05-26T01:32:00.000+01:002006-05-26T01:55:05.853+01:00CherryPy is Gone!When I posted my last missive, my version of Crunchy didn't do everything it should have - but I've now got it to a point where everything apart from exiting works (I've decided to drop support for arbitrary local tutorials for now - see below). So I can now say that CherryPy has been eliminated without any major issues.<br /><br />The problem with exiting seems to be because HTTP/1.1 has persistent connections, so BaseHTTPServer.HandleRequest never actually returns (I don't really understand the intricacies of HTTP very well though - I should go and read the RFC I guess). I've given the code to Andre - maybe he can spot something to help.<br /><br />I have some ideas about packaging groups of individual tutorials as "courses" - with additional metadata describing the relationships between tutorials (pages) in the course (see Andrew Harrington's <a href="http://mail.python.org/pipermail/edu-sig/2006-May/006496.html">post</a> on the edu-sig list - this is what prompted the idea).<br /><br />Basically, a course would be encapsulated in a directory (could also some kind of archive, say a zip file), and would have an index file that contained any metadata needed in multiple pages. The individual pages would then be referenced by the index (naturally pages could still link to each other). The index could also define some bijection from page "meta-names" to page files.<br /><br />I will post my ideas on the edu-sig list once they're more mature.Johannes Woolardhttp://www.blogger.com/profile/11444047692360756346noreply@blogger.com0tag:blogger.com,1999:blog-28236292.post-1148480524848584582006-05-24T15:15:00.000+01:002006-05-24T15:22:53.193+01:00Progress so farI'm a naturally impatient person, so I actually started working on the project on Monday - 48 hours before I actually got confirmation I was in.<br /><br />So far I've succeeded in removing the dependency on CherryPy using the BaseHTTPServer module from the Python base libraries. This was actually quite fun, because I've written a simple HTTP server in C before, but the Python version is far more elegant. Note that while I have replaced the whole server, it didn't actually mean using much more code - which really goes to show how unnecessary CherryPy is when you're doing something moderatley simple.<br /><br />I am told that Elementtree is in Python 2.5, so getting rid of it isnt a priority at all (thankfully).<br /><br />I've sent the code to Andre, so hopefully we'll see something like this in a release soon.Johannes Woolardhttp://www.blogger.com/profile/11444047692360756346noreply@blogger.com0tag:blogger.com,1999:blog-28236292.post-1148463131784521952006-05-24T10:18:00.000+01:002007-05-09T11:54:18.490+01:00Hi!<span style="font-weight:bold;">Note:</span> this was written back in 2006 when I was about to become an SoC student, see <a href="http://pytute.blogspot.com/2007/04/and-back-again.html">this post</a> for some updates on what I'm doing now.<br /><br /><br />Hi, I'm Johannes Woolard, a 19 year old student at the University of Oxford in the UK. I've just heard that I've been accepted into the Google Summer of Code scheme to work for the Python Software Foundation. I'm planning to use this blog to document my progress and experiences along the way.<br />Briefly, my project involves writing interactive web-based tutorials for 7-14 year olds. I'm being mentored by Andre Roberge (he has a <a href="http://aroberge.blogspot.com">blog</a> too).Johannes Woolardhttp://www.blogger.com/profile/11444047692360756346noreply@blogger.com3