Showing posts with label python. Show all posts
Showing posts with label python. Show all posts

Thursday, May 31, 2012

Python: Killing subprocesses on Windows

We're on Python 2.6, and using the subprocess module to execute some applications (like we do on our build system), sometimes things hang (i.e. an open dialog box or something).

So far, I've just killed the process after timeout, which is fine if there are no childprocesses in there.

Now, we've shifted to the Nose module for running our unit-tests (and Freshen for the specification-by-example acceptance tests), and that always uses one or more subprocesses to execute tests in... ergo: problems.

The simplest way is just to run taskkill, which is available at least from Windows XP...

p = subprocess.POpen(....)
# wait for exit or timeout
if timeout:
  subprocess.call(['taskkill', '/F', '/T', '/PID', str(p.pid)])

If you don't want to run another program, there's the long way around, by using comtypes and ctypes to access the WMI and Win32 API functions. I wrote this before I found out about the above, but it was easy to port C++ samples to this, and I already had comtypes in our system. Maybe it's useful for someone...

def killsubprocesses(parent_pid):
    '''kill parent and all subprocess using COM/WMI and the win32api'''
    
    log = logging.getLogger('killprocesses')
    
    try:
        import comtypes.client
    except ImportError:
        log.debug("comtypes not present, not killing subprocesses")
        return
    
    logging.getLogger('comtypes').setLevel(logging.INFO)
    
    log.debug('Querying process tree...')

    # get pid and subprocess pids for all alive processes
    WMI = comtypes.client.CoGetObject('winmgmts:')
    processes = WMI.InstancesOf('Win32_Process')
    subprocess_pids = {} # parent pid -> list of child pids
    
    for process in processes:
        pid = process.Properties_('ProcessID').Value
        parent = process.Properties_('ParentProcessId').Value
        log.trace("process %i's parent is: %s" % (pid, parent))
        subprocess_pids.setdefault(parent, []).append(pid)
        subprocess_pids.setdefault(pid, [])
      
    # find which we need to kill
    log.debug('Determining subprocesses for pid %i...' % parent_pid)

    processes_to_kill = []
    parent_processes = [parent_pid]
    while parent_processes:
        current_pid = parent_processes.pop()
        subps = subprocess_pids[current_pid]
        log.debug("process %i children are: %s" % (current_pid, subps))
        parent_processes.extend(subps)
        processes_to_kill.extend(subps)

    # kill the subprocess tree
    if processes_to_kill:
        log.info('Process pid %i spawned %i subprocesses, terminating them...' % 
            (parent_pid, len(processes_to_kill)))
    else:
        log.debug('Process pid %i had no subprocesses.' % parent_pid)

    import ctypes
    kernel32 = ctypes.windll.kernel32
    for pid in processes_to_kill:
        hProcess = kernel32.OpenProcess(PROCESS_TERMINATE, FALSE, pid);
        if not hProcess:
            _log.warning('Unable to open process pid %i for termination' % pid)
        else:
            _log.debug('Terminating pid %i' % pid)                        
            kernel32.TerminateProcess(hProcess, 3)
            kernel32.CloseHandle(hProcess)

This code could of course be improved by not walking all processes, OTOH, I have between 80 and 200 running on my PC at all times, so it's not a lot of data. Especially, since I only call this function if I've already waited too long, it doesn't matter if it takes a few more seconds.

Also, I long for Haskell where this tree-walking mapping stuff could be way more concise. Maybe python's itertools package has something I could use to avoid coding the dirty stuff by hand.

A few more solutions is available at this StackOverflow question, depending on what kind third-party of libraries you have available.

Moral of the story:

It doesn't matter if you have to write a lot of code in order to find out that there's a one liner which solves your problem. Use the short version and be happy you don't have to maintain the 30+ lines version. :)

Monday, January 17, 2011

Configuring PyQt across unit-tests

We had some problems with configuring the PyQt API version across unit-tests. The problems arise because several modules try to call sip.setapi, which you can only do once (even if you call it with the same args, which ought to be idempotent, you'd think.)

Anyway, what I did was to create a module called 'pyqtconfig' with the following contents:


import sip
sip.setapi('QString', 2)
sip.setapi('QVariant', 2)
sip.setapi('QTextStream', 2)


import PyQt4.Qt
Qt = PyQt4.Qt

Then I simply do 'from pyqtconfig import Qt' in each module. Works like a charm.

(If you want to import each module separately, you need to adapt the above code somewhat. I'll leave that as an exercise for the reader. :)

Sunday, January 9, 2011

BuildBotIcon - Python Style (and it's got tests this time!)

If you've read earlier posts here, you might know that I'm behind BuildBotIcon, a small Java application that sits in the system tray, polling one or more BuildBot servers and reports any changes. It also plays sounds and turns lava lamps on or off.

I've decided to do a version in Python using PyQt instead. It's mostly as a fun exercise swim in the ecosystem of python (setuptools, unittest, mocking, coverage, yaml, etc) but it might come out better than the original.

PyQs nice, and the new-style signals-slots makes it a breeze to do what you want if, assuming you know your way around Qt. (Not that it's hard to learn, I just knew Qt pretty well before embarking on this project.)

Writing unittests are a breeze, and I think that it sure helps knowing that the code works, instead of knowing it only compiles. Coming from C++ this is actually pretty nice change. However, you should write with testing from the beginning. Grafting unit-testing later on is difficult and might not be a net win for all components, especially in C++ . (Mostly since you've run them for a while and the obvious (and thus easy-to-test-for) bugs have already been ironed out.)

However, the greatest fun so far has been using Mock, yet another mock object module for python that takes a nice and intuitive approach to mocking, IMHO. Instead of record/replay, or setup/run/verify, it's more like mock/run/assert. Read Mocking, Patching, Stubbing: all that Stuff (and it's link to Mocks aren't Stubs) for more information on this.

The ability of substituting any function or object in the APIs you're running makes unittesting immensely powerful and thorough. If you trust the framework (Qt in my case) it's very easy to verify that play() is called on the right QSound (without making noises) or that get() is run with the right URL to access the network.
Of course, more complex behaviours are trickier to mock properly, so it helps to refactor your app into testable parts (which you should do anyway).

Also, I've used YAML for the settings file this time (using PyYAML). All is good and well, except that it turns out that writing the settings file first and implementing the app second had probably been a better idea. TDD again.

It has all fallen out pretty nicely so far, except that, similar to the Java version, I've bungled together the backend that does networking and parsing with the UI. Since I currently have three ways to show build state (tray icon, sound and lava lamps) I should really had put a Listener interface in there. Fixing that now means refactoring the unit tests too, which is less fun. :-|

Nevertheless, all of this is experience is bound to pay back in my ThrustRacer-project (should I get it off the ground) or at work, where we're about to embark on a year of much Python coding. I don't expect either to get it right from the start, one never does. :)

Saturday, November 27, 2010

Mercurial extension writing

I just spent the better part of the evening and early night to hack together a Mercurial extension to check for and fix tabs and trailing whitespace in files to be committed.

Diving into a (relatively) poorly documented Python API is not easy. My previous attempts with Django (extremely good docs) and BuildBot (perfectly acceptable docs) went much better in terms of tackling the early learning curve.

As an example, I have not yet managed to figure out how to get the added diff lines of the working directory, or the files included in an upcoming commit (this are slightly different). The MercurialAPI wiki page only gives a brief overview, after that you're on your own and have to use the source. This is pretty fine, except that dynamic languages need more written docs if the source is to be understood better.

Anyway, the extension works pretty nicely as I decided to check the _entire_ file instead of just the changed lines. Shouldn't make a big difference performance wise, and it will allow a code base to go from mixes-tabs to spaces-only much faster. I can always revisit that part later if the need arises.

The next part in the plan is to write a small script that auto-injects this into each developers hgrc file on checkout. (I have a script that one must run, more or less, to deploy/unzip third-party libraries and set up environment variable, so it can just as well edit the hgrc file too.) I'm not worried about the security implications currently, as we're four developers on a corporate network, so everyone should play nicely at the moment, and secondly, I can always safeguard the paths to the extension (or put in in a subrepo) to which no-one but a select few, including myself, have commit/push rights (in case of a rogue developer).

The next little script I write will probably be something like that, to allow deployment of local extensions. I'll also add some extra meta-data to a commit and a central-repo hook to check for that, so that one can't push without the meta-data (ensuring that everyone starts using the hook ASAP) and also run the same checks there (in case someone decides to override the hook).

Hacking should not happen out of pure malice, but I've had developers disabling the global warnings-in-log-at-exit popup just because they where annoyed by them. (The best way would've been to fix the warnings inside the app, or at least have a chat with me on which log messages are warnings and if it's worthwhile to add a feature where said developer may disable the popup for a few hours.) Since the popup was disabled, many others in the team missed warnings or errors in the log and subsequently the CI build failed with warnings after commit. Sadly quite contraproductive and against the spirit of early-warnings.

Summing it up, I really enjoy hacking together small things in Python, but for larger projects it seems to become pretty unbearable and you really want to know the types of the variables. I really wish C# and WPF was available on more than one platform, so that we could go with that instead of C++/Python on Qt. It does seem to be a nicer road to walk.

(Yes, I've thought about Java, and I've worked with it for a few years. And no, I don't really like it, but I suppose SWT is at leats a bearable GUI-API (i.e. better than Swing), but lately with the Oracle takeover I'm very wary of betting on Java going anywhere but down the drain for non-enterprise projects.)

Obligatory links:

Friday, January 22, 2010

Validating DTDs in python with lxml

After replacing the string-pasting in Buildbot's web-backend with proper html-templates (using Jinja), I decided to write a unit-test that follows all links on all pages and make sure they validate and don't contain stale links.

lxml seemed like a nice tool for that, but our XHTML starts with a reference to w3.org's DTD I got this straight away:
failed to load HTTP resource
Googling found me the Caching DTDs using lxml and etree article by Jimmy Stratton. He had ran into the same problem last fall, dug around for a while (against the forces of nature, in this case represented by authors and documentation for lxml and python), and was nice enough to post his solution on the web.

The resolver works flawlessly. Thanks Jimmy! I owe you one.

Update: Since this is a reasonably popular post, here's the unit test for BuildBot, which applies this using the Twisted framework to validate all linked-to pages on the website. (It's was lost from the source tree in one of the many refactorings and cleanups of BuildBot's test suite. Also, it added some troublesome dependencies for the many builds slaves, and was too broad in scope. It should be re-cast as unit-tests for each page's possible states.)

Sunday, December 14, 2008

Mercurial and python 2.6! :)

This is a follow up on my previous post 'Distributed Version Control Systems on Windows' where I just want to note that I've made some progress with Mercurial by using Python 2.6 instead of 2.5.2.

The good thing is that Python 2.6 uses MSVS 2008, which I do have available. So with that, I can use setuptools to compile python libs. Including Mercurial!

Compiling and install Mercurial into my Python 2.6 64-bit installation was dead easy, by the books all the way! Yay!

That will probably help with running buildbot/mercurial/trac on a pure windows-based server (as in no cygwin). But I'm considering using a linux box for this anyway, since it's smoother and more easy to add misc minor stuff to. (Not disregarding setting up ssh-accounts for mercurial users, which is what I've done currently.)

I'm currently using it to try to assist the Mercurial situation on BuildBot, which is lacking a bit, especially when it comes to in-repo branches. There are patches on the buildbot trac, but they haven't been applied and the crucial one won't be added until it gets a proper test-case.

So, I'm working on that. There's some issue with line-endings preventing try-patches to apply cleanly in the tests. (It works in real life, as I did use the try-functionality at my last job. Never got it ready for every developer though...)

Monday, November 24, 2008

Python - list clearing?!!

Today's trick question: How do you clear a list in python?

Not by .clear(), as you do with other python containers set or dicts (or _any_ in C++/Java/whatnot).

Nonono, you assign the entire slice of the list to the empty list, or delete the whole slice!

mylist[:] = []
del mylist[:]


.. .. .. .. Arrghh!!!!

Why do I have to learn the slice syntax, and the fact that there is a slice of the whole pie, and that I can assign a list to a slice, just to clear the list?!

Appending, concatenating and clearing are simple & common ops, that should be represented clearly and simply. Slicing is advanced, and when I want to do slicing, I read up on it, but I should need it for that. It's like requiring someone to learn discrete integration just to do addition; valid but only correct in the very bad technically-correct kind of way.

(And no, just assigning the empty list to it does not work, as that rebinds the name, it does not change the original list. I have no issue with that, as it's all pointers/references everywhere.)

Apparently, they haven't fixed this in Py3k either.

Looking through forums where this appear every month since Python was first created, the sentiment seems to be that since you can do it with slicing, there's no need for another way to do it!

I began to understand why people feel that the python community is regarded as a bit 'frosty' and cold.

I still like python a lot, but ... Bah!