Validating DTDs in python with lxml

After replacing the string-pasting in Buildbot's web-backend with proper html-templates (using Jinja), I decided to write a unit-test that follows all links on all pages and make sure they validate and don't contain stale links.

lxml seemed like a nice tool for that, but our XHTML starts with a reference to w3.org's DTD I got this straight away:

failed to load HTTP resource

Googling found me the Caching DTDs using lxml and etree article by Jimmy Stratton. He had ran into the same problem last fall, dug around for a while (against the forces of nature, in this case represented by authors and documentation for lxml and python), and was nice enough to post his solution on the web.

The resolver works flawlessly. Thanks Jimmy! I owe you one.

Update: Since this is a reasonably popular post, here's the unit test for BuildBot, which applies this using the Twisted framework to validate all linked-to pages on the website. (It's was lost from the source tree in one of the many refactorings and cleanups of BuildBot's test suite. Also, it added some troublesome dependencies for the many builds slaves, and was too broad in scope. It should be re-cast as unit-tests for each page's possible states.)

Dirty coding tricks

If you've gotta ship, you've gotta ship.

See this Gamasutra article for some funny stories. :)

Teh build is fail!

Just to complete the gallery:


The responsible one has left the building but it's been pretty effective so far in getting people to fix the build a.s.a.p. :)

Also, the latest version of BuildBotIcon has optional sound, and currently we have Homer Simpson going 'DOH!' or 'Woohoo!' (rather loud). This makes everyone aware of it instantly.

(And yes, I've been reading lolcats too much lately..)

Oh noes, the red light is on!

This is how it looks before things get really bad. The build was fixed before the lamp was running at full speed though. :)

Lava lamps - up and running!

One picture, 1k words, etc:



We've also added a timer to turn them off during the night, along with the status monitor (which you can't see), to be nice to the environment.

Again, the USB board can be bought from Active Robots who were very friendly and helpful shipping the package to me, in spite of the efforts of our local post office. ;)

Lava lamp USB board assembly pictures

Here's some pictures of the assembly process. It went reasonably well, but making sure you have big enough drills beforehand helps a bit. :)

Here's how the board looks before assembly:



First, check that the circuit board fix in plastic box. Tight fit. :)



Drill holes for 230V mains (in/out) and USB cable: (Had to use a knife to make the hole for the USB-B connector fit, as it is slightly asymmetric. Luckily, the plastic was very easy to work with.)



Drilled for circuit board with USB connector in place (avoids misalignment):



Prepare cables:



Do the wiring (and remember to pull the cables through the holes in the box first):



Fit the wired circuit board in the box, make sure all cables go inside and put tie-wraps on the cables to secure against mechanical stress:



Here's the box with the lid on:



Ta-daaa. :)

As you can see, I currently only use one relay. It's of course possible to use all four, but that requires more drilling and wiring, and more lava lamp shelf space. :)

Always leave room for upgrades, I say. ;)

Lava lamp - 99.5%

Here's a tech-porn picture. :)

It's connected at office and it works, but I didn't have time for lavalamps.

Instead, I connected it to my office stereo system and set the computer to constantly play something from Tribute to Vengaboys when the build has failed.

Forget the carrot, it's all about the stick. :)