Thursday, May 31, 2012

Python: Killing subprocesses on Windows

We're on Python 2.6, and using the subprocess module to execute some applications (like we do on our build system), sometimes things hang (i.e. an open dialog box or something).

So far, I've just killed the process after timeout, which is fine if there are no childprocesses in there.

Now, we've shifted to the Nose module for running our unit-tests (and Freshen for the specification-by-example acceptance tests), and that always uses one or more subprocesses to execute tests in... ergo: problems.

The simplest way is just to run taskkill, which is available at least from Windows XP...

p = subprocess.POpen(....)
# wait for exit or timeout
if timeout:
  subprocess.call(['taskkill', '/F', '/T', '/PID', str(p.pid)])

If you don't want to run another program, there's the long way around, by using comtypes and ctypes to access the WMI and Win32 API functions. I wrote this before I found out about the above, but it was easy to port C++ samples to this, and I already had comtypes in our system. Maybe it's useful for someone...

def killsubprocesses(parent_pid):
    '''kill parent and all subprocess using COM/WMI and the win32api'''
    
    log = logging.getLogger('killprocesses')
    
    try:
        import comtypes.client
    except ImportError:
        log.debug("comtypes not present, not killing subprocesses")
        return
    
    logging.getLogger('comtypes').setLevel(logging.INFO)
    
    log.debug('Querying process tree...')

    # get pid and subprocess pids for all alive processes
    WMI = comtypes.client.CoGetObject('winmgmts:')
    processes = WMI.InstancesOf('Win32_Process')
    subprocess_pids = {} # parent pid -> list of child pids
    
    for process in processes:
        pid = process.Properties_('ProcessID').Value
        parent = process.Properties_('ParentProcessId').Value
        log.trace("process %i's parent is: %s" % (pid, parent))
        subprocess_pids.setdefault(parent, []).append(pid)
        subprocess_pids.setdefault(pid, [])
      
    # find which we need to kill
    log.debug('Determining subprocesses for pid %i...' % parent_pid)

    processes_to_kill = []
    parent_processes = [parent_pid]
    while parent_processes:
        current_pid = parent_processes.pop()
        subps = subprocess_pids[current_pid]
        log.debug("process %i children are: %s" % (current_pid, subps))
        parent_processes.extend(subps)
        processes_to_kill.extend(subps)

    # kill the subprocess tree
    if processes_to_kill:
        log.info('Process pid %i spawned %i subprocesses, terminating them...' % 
            (parent_pid, len(processes_to_kill)))
    else:
        log.debug('Process pid %i had no subprocesses.' % parent_pid)

    import ctypes
    kernel32 = ctypes.windll.kernel32
    for pid in processes_to_kill:
        hProcess = kernel32.OpenProcess(PROCESS_TERMINATE, FALSE, pid);
        if not hProcess:
            _log.warning('Unable to open process pid %i for termination' % pid)
        else:
            _log.debug('Terminating pid %i' % pid)                        
            kernel32.TerminateProcess(hProcess, 3)
            kernel32.CloseHandle(hProcess)

This code could of course be improved by not walking all processes, OTOH, I have between 80 and 200 running on my PC at all times, so it's not a lot of data. Especially, since I only call this function if I've already waited too long, it doesn't matter if it takes a few more seconds.

Also, I long for Haskell where this tree-walking mapping stuff could be way more concise. Maybe python's itertools package has something I could use to avoid coding the dirty stuff by hand.

A few more solutions is available at this StackOverflow question, depending on what kind third-party of libraries you have available.

Moral of the story:

It doesn't matter if you have to write a lot of code in order to find out that there's a one liner which solves your problem. Use the short version and be happy you don't have to maintain the 30+ lines version. :)

Tuesday, April 17, 2012

E-books are awesome! (And the Kindle app is ok-ish)

I've recently read quite a few books on my Android tablet (Asus Eee Pad Transformer) via the Kindle app, and the experience has been quite exquisite so far.
  • Quiet reading at night (in the dark) is no problem at all. Excellent for an insomniac as yours truly when the rest of the family is sleeping soundly in the same room.
  • Start/stop time is virtually nil, since the app remembers the last page read.
    (This is also synced between devices, so reading a few pages off the phone on the train/tram/bus/loo/whatnot works well.)
  • The Oxford dictionary is included, so most odd words can be explained quickly.
  • No bookshelves required!
The drawbacks are not insignificant:
  • not all books are good to read in night mode, as they don't cope with the inverted color scheme.
  • maps/drawings are seldom included or look good. The format & applications are best suited to text-only books.
  • night-mode makes everything feel doomed and evil. Do bear this in mind  when reading Lovecraft, The Road or other horror-themed literature.
  • table of contents and chapters are ususally poorly implemented and could benefit greatly from a little bit more adaptation to the electronic format.
Apart from these issues, it's pretty excellent for fiction and beats trying to hold a regular old paper book hands down, especially as I read mostly during night.

What's more worrying in the long term is perhaps the DRM bit (I can't really lend or sell an e-book currently) as well as Amazon's monopoly/monopsony.

This is better explained in this excellent blog post. 

Monday, March 26, 2012

Status update

My main occupation these days is as executive of damage control, for a 1y old toddler.  Doing so leaves precious little time and brain power for any creative software work.

Whatever free time I get I tend to use for gaming/surfing, which is what my spare brain cycles allows for. It's plain hard to  regularly find some dedicated time for creative work, when your brain is trying to cope with the mood swings and antics of someone who has to program her neural network by trial and error. :)

So, the going-ons in my software world (outside work) are rather slim:
  • BuildBot is ticking along fine without me, although I'm trying to do a bit of pull request checking, since  Dustin Mitchell, "chief maintainer", needs some offloading on that part.
  • ThrustRacer remains a concept and dream. I've had some doubts as to whether twitchy arcade games can be as good on mobile devices as they were in the golden age of the Tac-2, thus wondering if I should spend my time thinking on thinking about something else instead.
  • TF2 is fun, but kinda hard unless you find a server with comparable players. Ranking, seeding and matching is sorely needed. I had to by a new high-DPI mouse to be at least vaguely competitive
  • Skyrim is .. scary .. and buying the discount PC-version on Steam over the overprices PS3 version was not as good an idea as it first sounded. I need a new graphics card, which costs way more! (OTOH, might help with TF2). The scariness (i.e. I'm really afraid to see what will come at me next...) is a problem though, since I play at night/evenings (ref. above mentioned toddler) and playing in the dark, quiet and trying to quickly go asleep afterwards doesn't really mix well. (Still haven't finished BioShock, nor seen Alien 1 to completion...)
  • Our (recently acquired) house sorely needs more power and Ethernet outlets, a small herd of Ardunio boards controlling all the light fixtures and a motorized garage door.

Ah, the woes of geekdom and fatherhood combined...