Blog

Python ‘pip’ and Windows registry corruption

Background

In my recent post about setting up a 64 bit Windows Django development environment, I described an initial problem with Virtualenv which then led me through a whole series of problems and fixes and finally to what I thought was a working 64 bit Python development environment. However, I recently realised that the original underlying issue still remained, such that any attempt to install a new package using pip would result in failure due to that same error message.

For example, when I tried to install Beautiful Soup via pip, I got the following error:

C:\> pip install beautifulsoup4
Downloading/unpacking beautifulsoup4
Cleaning up...
Exception:
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\pip\basecommand.py", line 122, in main
status = self.run(options, args)
File "C:\Python27\lib\site-packages\pip\commands\install.py", line 278, in run

requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundl
e=self.bundle)
File "C:\Python27\lib\site-packages\pip\req.py", line 1229, in prepare_files
req_to_install.run_egg_info()
File "C:\Python27\lib\site-packages\pip\req.py", line 292, in run_egg_info
logger.notify('Running setup.py (path:%s) egg_info for package %s' % (self.s
etup_py, self.name))
File "C:\Python27\lib\site-packages\pip\req.py", line 265, in setup_py
import setuptools
File "C:\Python27\lib\site-packages\setuptools\__init__.py", line 12, in
from setuptools.extension import Extension
File "C:\Python27\lib\site-packages\setuptools\extension.py", line 7, in
from setuptools.dist import _get_unpatched
File "C:\Python27\lib\site-packages\setuptools\dist.py", line 15, in
from setuptools.compat import numeric_types, basestring
File "C:\Python27\lib\site-packages\setuptools\compat.py", line 19, in
from SimpleHTTPServer import SimpleHTTPRequestHandler
File "C:\Python27\lib\SimpleHTTPServer.py", line 27, in
class SimpleHTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
File "C:\Python27\lib\SimpleHTTPServer.py", line 208, in SimpleHTTPRequestHand
ler
mimetypes.init() # try to read system mime.types
File "C:\Python27\lib\mimetypes.py", line 361, in init
db.read_windows_registry()
File "C:\Python27\lib\mimetypes.py", line 263, in read_windows_registry
with _winreg.OpenKey(hkcr, subkeyname) as subkey:
TypeError: must be string without null bytes or None, not strCode language: Access log (accesslog)

Registry Problems

My initial research led me to believe the problem was due to missing registry entries that were required by ‘setuptools’ or ‘pip’ on 64 bit Windows systems. However, my attempts to follow the recommended registry fixes did not help at all. Neither did installing native 64 bit builds of ‘setuptools’ and ‘pip’ – even after successfully installing the 64 bit versions, they still failed to install any packages with exactly the same error.

I finally found a helpful discussion thread that led me to investigate the ‘mimetypes.py’ standard Python module, which is the origin of the traceback I keep getting. There was a suggestion that perhaps a corrupted registry key in ‘HKEY_CLASSES_ROOT’ could be breaking the ‘read_windows_registry()’ function.

I wrote the following script (for Python v2.7.6) to identify any problematic registry keys:

import _winreg as winreg

hkcr = winreg.OpenKey(winreg.HKEY_CLASSES_ROOT, '')

for i in range(999999):
    try:
        key = winreg.EnumKey(hkcr, i)
        if '\0' in key:
            print "Found null character in key: HKEY_CLASSES_ROOT/%s" % str(key).replace('\0', '*')
    except WindowsError:
        print "Finished!"
        break
Code language: Python (python)

This identified around 10 problematic keys which contained trailing null characters.

Alternative Solutions

Remove Corrupted Keys

The Windows Sysinternals RegDelNull tool can be used to easily find and delete any registry keys containing null characters. Although the problematic keys are being displayed in ‘HKEY_CLASSES_ROOT’, in fact they are being pulled in from ‘HKEY_LOCAL_MACHINE’ so that is where we must search:

F:\> RegDelNull.exe hklm -s

RegDelNull v1.10 - Delete Registry keys with embedded Nulls
Copyright (C) 2005-2006 Mark Russinovich
Sysinternals - www.sysinternals.com

Null-embedded key (Nulls are replaced by '*'):

HKLM\SOFTWARE\Classes\{03223D4D-1B28-4325-9A96-9C5A4C8EA8BC}*

Delete? (y/n)Code language: Access log (accesslog)

If you are happy to completely remove any problematic keys, and accept any associated breakages that may occur, then this is the easiest solution.

Manually Patch ‘mimetypes.py’

I applied a manual 3 line hack-patch to the Python v2.7.6 ‘mimetypes.py’ module (around line 260) as follows:

with _winreg.OpenKey(_winreg.HKEY_CLASSES_ROOT, '') as hkcr:
    for subkeyname in enum_types(hkcr):
        try:
            if '\0' in subkeyname: # new
                print "Skipping bad key: %s" % subkeyname # new
                continue # new
            with _winreg.OpenKey(hkcr, subkeyname) as subkey:Code language: Python (python)

With these 3 extra lines in place, I am now able to install new packages using ‘pip’ without any errors at all. This is potentially a good workaround for those who don’t want to edit their registry, although it is very nasty to hand-edit your core Python installation in this way. For the record, this is still an issue with the latest build of Python v3.4. Although this is caused by problematic registry entries, it would be nice if Python handled the situation more gracefully.

Update – 14th June 2015

This issue has since been reported in a couple of official Python bugs:

It looks like the issue has now been patched in both the Python 2.7 and 3.4 branches, although the status of Issue 22028 remains “open” so I don’t believe the fix is generally available quite yet.

Tags: , , , , ,

This Post Has 34 Comments

  1. mookrs 26 April 2014 Reply

    Awesome! This post helped me fix the problem. Thank you very much!

    1. Ian Thomas 26 April 2014 Reply

      That’s excellent – thanks very much for letting me know! Ian

  2. Cheng Yen-Yu 30 April 2014 Reply

    “Windows Sysinternals RegDelNull tool” is work for me,thank you.

    1. Ian Thomas 30 April 2014 Reply

      Excellent – thanks for the feedback! Ian

  3. Tom 2 July 2014 Reply

    Thank you very much for the info. I’m curious as to why this was so hard to find, yet the problem is very game-breaking. Is there anywhere else that talks about this same issue?
    Cheers.

    1. Ian Thomas 2 July 2014 Reply

      Hi Tom,

      Thanks for the feedback – I’m glad this was helpful. I’ve no idea why this issue is not better documented. I couldn’t find any decent information or any clear recommended solutions, so I gathered together all the separate nuggets of information I could find and came up with my own workaround (albeit not a very elegant one!). I’m getting quite a few hits on this post so I’m guessing others are also struggling to find proper solutions elsewhere.

      Cheers, Ian

  4. Pete 21 July 2014 Reply

    Thanks for the help, been looking for ages for a solution. Curiously only changing the mimetypes.py module worked, and popped up with 12 keys it skipped, yet when i ran the regdelnull hklm -s, it couldn’t find any.

    Cheers 😀

    1. Ian Thomas 21 July 2014 Reply

      Hi Pete,

      Thanks very much for the feedback – I’m really glad this article was helpful! I’m not too sure why ‘regdelnull’ didn’t track those problematic keys down, but at least the ‘mimetypes.py’ hack got things working for you.

      Cheers, Ian

  5. Joben Rara 23 August 2014 Reply

    Thank you. This has saved me a ton of grief. It’s still a problem in 2.7.8. Why not submit as a patch?

    1. Ian Thomas 23 August 2014 Reply

      Thanks for the feedback – glad this helped you. Perhaps I should submit a patch! 🙂

  6. Johannes Kraemer (@johanneskraem3r) 23 August 2014 Reply

    Having the same exact issue. Changing mimetypes.py makes it work. I just tried for a few hours to remove the bad registry key without success. RegDelNull.exe does not find anything, but your python tool finds a bad key. (This is on Windows 8) . If anyone has an idea how to remove the bad key from my registry let me know. @Ian: If you can, submit a patch! This cost me a few hours today. Thanks again for posting this.

    1. Ian Thomas 23 August 2014 Reply

      Hi Johannes,

      Thanks very much for the feedback – glad to help another satisfied customer! I’m not sure how to remove the bad key if ‘RegDelNull.exe’ won’t do it for you. I’ve never submitted a patch to Python itself before, but I’ll see if I can get anywhere with it…

      Thanks again, Ian

  7. magnet 16 September 2014 Reply

    Sure enough. Searched the net for a week and I finally found your blog! :] Thanks man. I had tried everything except hacking mimetypes.py myself and your workaround is the only thing that got me working so, to me it’s quite elegant! Keep up the good work and thanks again!

    1. Ian Thomas 16 September 2014 Reply

      Thanks very much for the feedback – I’m really glad this “elegant” patch helped you! 😉

      Ian

  8. Clare Macrae 15 October 2014 Reply

    Hi Ian!

    This saved the day for me… thanks!

    Just wanted to point out a typo in the code you quote for ‘mimetypes.py’ – the last line – ‘with _winreg.OpenKey(hkcr, subkeyname) as subkey:’ – needs to be indented 4 more spaces, to be consistent with the original code in the file – i.e. the 2nd ‘with’ needs to line up with the ‘if’

    Thanks again! Clare

    1. Ian Thomas 17 October 2014 Reply

      Hi Clare,

      Thanks very much for the correction – I’ve added the missing spaces! Glad this post was helpful.

      Ian

      1. Luca 19 April 2015 Reply

        Dear Ian, dear Clare,
        Thanks for providing this solution

        1. Ian Thomas 19 April 2015 Reply

          You’re very welcome – thanks for the feedback!

  9. Tobias T 8 February 2015 Reply

    Thanks very much for this post, it saved me a lot of headache. I actually found it because I was searching for the offending key “03223D4D-1B28-4325-9A96-9C5A4C8EA8BC”, the exact same as in your RegDelNull output. Do you happen to have any idea which application could be the origin of those keys?

    1. Ian Thomas 8 February 2015 Reply

      Hi Tobias,

      Thanks very much for the feedback – I’m glad this post was helpful! Sorry but I have no idea which application is responsible for that particular key. That’s why I felt it was safer to ignore the null keys rather than delete them, because I don’t know which applications may break if I simply remove them all. This issue is a real pain!

      Ian

    2. Ian Thomas 22 February 2015 Reply

      Tobias – I have since discovered that some of my invalid registry keys may have been created by a software package called “FastPictureViewer Codec Pack”. Do you have that also? Ian

      1. Tobias 23 February 2015 Reply

        Yeah, that could be it.

  10. Mike Trotman 17 March 2015 Reply

    This blog entry was a lifesaver!

    I hit this problem when trying to run the AWS S3 CLI to backup files from my Windows PC to Amazon S3.
    Running regdelnull.exe cleared Nulls from about 5 registry keys and solved the problem

    Like Ian Thomas I also have FastPictureViewer installed – so this may be causing the problem – but I did not record and trace the deleted keys so can’t help clarify further.

  11. Ben 9 April 2015 Reply

    Just. Thanks.
    🙂

  12. Dave Kielpinski 26 May 2015 Reply

    Thanks – that was tremendously helpful! Used your script to find the keys, then RegDelNull to remove them.

    1. Ian Thomas 26 May 2015 Reply

      That’s excellent – thanks very much for the feedback!

  14. xgiovio 2 December 2016 Reply

    Thank you very much. Resolved in minutes. Had problem with rbt tool and embedded python 2.7.

Leave a Reply