Saturday, November 27, 2010

Mercurial rebase/merge musings part 2

In a previous post I wrote about how nice it'd be if Mercurial automatically could choose rebase or merge depending on whether there are conflicts that need manual resolution.

So, "yesternight", after finishing the checkfiles extension I wrote about recently I couldn't go back to sleep (despite it being 2:30 AM). So, I got back up and hacked together the rebaseif extension that adds a command and an option to pull that does exactly that. It turns out it wasn't that hard, although the process could probably be optimized slightly as it currently attempts a rebase, aborts on conflicts and then does a merge. (The better way would be to detect which operation would succeed and perform that.)

Now, some support for that in TortoiseHg and we're all set. :)

Mercurial extension writing

I just spent the better part of the evening and early night to hack together a Mercurial extension to check for and fix tabs and trailing whitespace in files to be committed.

Diving into a (relatively) poorly documented Python API is not easy. My previous attempts with Django (extremely good docs) and BuildBot (perfectly acceptable docs) went much better in terms of tackling the early learning curve.

As an example, I have not yet managed to figure out how to get the added diff lines of the working directory, or the files included in an upcoming commit (this are slightly different). The MercurialAPI wiki page only gives a brief overview, after that you're on your own and have to use the source. This is pretty fine, except that dynamic languages need more written docs if the source is to be understood better.

Anyway, the extension works pretty nicely as I decided to check the _entire_ file instead of just the changed lines. Shouldn't make a big difference performance wise, and it will allow a code base to go from mixes-tabs to spaces-only much faster. I can always revisit that part later if the need arises.

The next part in the plan is to write a small script that auto-injects this into each developers hgrc file on checkout. (I have a script that one must run, more or less, to deploy/unzip third-party libraries and set up environment variable, so it can just as well edit the hgrc file too.) I'm not worried about the security implications currently, as we're four developers on a corporate network, so everyone should play nicely at the moment, and secondly, I can always safeguard the paths to the extension (or put in in a subrepo) to which no-one but a select few, including myself, have commit/push rights (in case of a rogue developer).

The next little script I write will probably be something like that, to allow deployment of local extensions. I'll also add some extra meta-data to a commit and a central-repo hook to check for that, so that one can't push without the meta-data (ensuring that everyone starts using the hook ASAP) and also run the same checks there (in case someone decides to override the hook).

Hacking should not happen out of pure malice, but I've had developers disabling the global warnings-in-log-at-exit popup just because they where annoyed by them. (The best way would've been to fix the warnings inside the app, or at least have a chat with me on which log messages are warnings and if it's worthwhile to add a feature where said developer may disable the popup for a few hours.) Since the popup was disabled, many others in the team missed warnings or errors in the log and subsequently the CI build failed with warnings after commit. Sadly quite contraproductive and against the spirit of early-warnings.

Summing it up, I really enjoy hacking together small things in Python, but for larger projects it seems to become pretty unbearable and you really want to know the types of the variables. I really wish C# and WPF was available on more than one platform, so that we could go with that instead of C++/Python on Qt. It does seem to be a nicer road to walk.

(Yes, I've thought about Java, and I've worked with it for a few years. And no, I don't really like it, but I suppose SWT is at leats a bearable GUI-API (i.e. better than Swing), but lately with the Oracle takeover I'm very wary of betting on Java going anywhere but down the drain for non-enterprise projects.)

Obligatory links:

Tuesday, November 9, 2010

New toy!

Also, I got a HTC Desire a few weeks ago. Love it!

It's really becoming the multi-purpose do-it-all device that we all need to make our life a non-stop information gathering/processing journey from the cradle to the grave. :-P

More on HTML & JavaScript Client Apps

In a recent post I wrote about the new log window for our Qt apps at work. Essentially, it delivers the log via AJAX to a web browser, with the html & javascript embedded in and served statically by the app. Works well enough.

However, there are two more ships on the horizon flying similar flags.

The first is my (secret) plan to build a debugger for a State Machine for a few years. Since Qt now has a state machine as of 4.6, and it seems both solid, well documented and behaves mostly correct (observing that we've yet to learn all ins and outs of a full state chart machine) is seems a good target for this. I've already written a QStateMachine to GraphViz translator, which outputs an SVG which can (with some good faith and heavy squinting) be interpreted as a UML state chart (cough cough).

Now, SVG is an XML document which has a DOM, which you can manipulate with Javascript in most modern browsers. Yay! However,  this did not, at the time of having the idea, include Google Chrome (my favorite), since SVG-DOM updates didn't redraw the image immediately.... They have fixed it since, but I am pretty busy with other things so this small show stopper held me up. I hit a bump, I move in another direction, especially when I'm just going somewhere to see where I end up. Will have to get back on this one ... :)

Anyway, for the final product, I think a nice event log and some dynamic colouring/resizes of states and transitions would work wonders for debugging the app. Breakpoints via mouse-click events in the graph is then the second step. Then there are some question marks, but the last step is, as always, profit. ;)

The problem is always to package it so that other people can re-use it. I have to try to get better at that, but it's sadly not on top of the priority pile when working at a startup.

Secondly, it seems as the BuildBot project is planning to boot my Jinja webstatus implementation that replaced the html-string-pasting in python. (I never blogged about that.. oh well.. fun times it was anyway) Anyways, sad as that can be, the new plan is for... a html/javascript-only client! The content can then be served statically (via twisted or your local "real" webserver, for more oomph) and the webpage then communicates with the buildmaster using relatively small AJAX requests.

I think the idea is pretty neat, and has much in common with the things above, so I might go ahead and tackle that as well. Or procastrinate until someone else does it and steal^h^h^hborrow ideas from that place. Anyways, the jinja-rewrite made it much easier to see the actual HTML, so the new javascript (preferrably using jQuery) client could build on top of the existing layout.

Small steps in the right direction makes the world a better place. :)

Also, some day in March, if things go as they should, the household will gain a new member. One very small and in need of almost-constant care. I estimate I will not be able to contribute to open source for a few years starting then. Best do what I can before that. ;)

Thursday, November 4, 2010

Mercurial rebase/merge musings

I asked about how-to-check-for-potential-merge-rebase-conflicts-in-mercurial on StackOverflow yesterday and got a few interesting responses.

I think I'm going to write a small plugin that rebases if there are no file conflicts, and merges otherwise. This is what I want myself and everyone else to do, as it helps with clearing up bad merges afterwards, and avoid "'rebase broke the build and my life sucks" situtations.

Update 2010-11-27: I've written an extension for this, see this post for details and links.

LogWindow 2.0 using LibQxt

I just implemented a small web server for our Qt apps that use LibQxt's QxtWeb classes to serve the application log over HTML.

Static files are served from QResources and log events are fetched by AJAX/jQuery, using a small javascript code to format them into a table in the web server.

Next on the todo-list is to add a dynamic filter by severity/thread/location. Should be pretty easy to get going.

We used an in-application LogWindow before, but that sucks performance wise and it doesn't redraw when debugging the GUI-thread, obviously. This is also way easier than creating and managing an logging application as an entirely different process.

The most recent LibQxt even has a JSONRPCService, which might've been useful. We're using a slightly older one so I manually implemented a WebService which serves JSON or html/css/js.

Anyway, this ought to be cleaned up a bit and contributed back to LibQxt.

Friday, January 22, 2010

Validating DTDs in python with lxml

After replacing the string-pasting in Buildbot's web-backend with proper html-templates (using Jinja), I decided to write a unit-test that follows all links on all pages and make sure they validate and don't contain stale links.

lxml seemed like a nice tool for that, but our XHTML starts with a reference to's DTD I got this straight away:
failed to load HTTP resource
Googling found me the Caching DTDs using lxml and etree article by Jimmy Stratton. He had ran into the same problem last fall, dug around for a while (against the forces of nature, in this case represented by authors and documentation for lxml and python), and was nice enough to post his solution on the web.

The resolver works flawlessly. Thanks Jimmy! I owe you one.

Update: Since this is a reasonably popular post, here's the unit test for BuildBot, which applies this using the Twisted framework to validate all linked-to pages on the website. (It's was lost from the source tree in one of the many refactorings and cleanups of BuildBot's test suite. Also, it added some troublesome dependencies for the many builds slaves, and was too broad in scope. It should be re-cast as unit-tests for each page's possible states.)