Welcome, guest | Sign In | My Account | Store | Cart

Python 2.4 provides the '-m' option to run a module as a script. However, "python -m <script>" will report an error if the specified script is inside a package.

Putting the following code in a module called "execmodule.py" and placing it in a directory on sys.path allows scripts inside packages to be executed using "python -m execmodule <script>".

Python, 62 lines
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
"""Run a module as if it were a file

Allows Python scripts to be located and run using the Python module namespace
instead of the native filesystem.
"""
from imp import find_module, PY_SOURCE, PY_COMPILED
import sys

__all__ = ['execmodule']

class _ExecError(ValueError): pass

def execmodule(module_name, globals=None, locals=None, set_argv0 = False):
    """Locate the requested module and run it using execfile

    Any containing packages will be imported before the module is executed.
    Globals and locals arguments are as documented for execfile
    set_argv0 means that sys.argv[0] will be set to the module's filename prior
    to execution (some scripts use argv[0] to determine their location).
    """
    if globals is None:
        globals = sys._getframe(1).f_globals # Mimic execfile behaviour
    if locals is None:
        locals = globals
    pkg_name = None
    path = None
    split_module = module_name.rsplit('.', 1)
    if len(split_module) == 2:
        module_name = split_module[1]
        pkg_name = split_module[0]
    try:
        # Import the containing package
        if pkg_name:
            pkg = __import__(pkg_name)
            for sub_pkg in pkg_name.split('.')[1:]:
                pkg = getattr(pkg, sub_pkg)
            path = pkg.__path__
        # Locate the module
        module_info = find_module(module_name, path)
    except ImportError, e:
        raise _ExecError(str(e))
    # Check that all is good
    module = module_info[0]
    filename = module_info[1]
    filetype = module_info[2][2]
    if module: module.close() # We don't actually want the file handle
    if filetype not in (PY_SOURCE, PY_COMPILED):
        raise _ExecError("%s is not usable as a script\n  (File: %s)" %
                          (module_name, filename))
    # Let's do it
    if set_argv0:
        sys.argv[0] = filename
    execfile(filename, globals, locals)

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print >> sys.stderr, "No module specified for execution"
    del sys.argv[0] # Make the requested module sys.argv[0]
    try:
        execmodule(sys.argv[0], set_argv0 = True)
    except _ExecError, e:
        print >> sys.stderr, e

Python 2.4 brings us the '-m' command line option to make it easy to run modules like the profiler or debugger as scripts (using "python -m profile <script>" and "python -m pdb <script>" respectively).

Unfortunately, -m is limited to top level modules only - it can't look inside packages. With execmodule in place, it becomes possible to do things like "python -m execmodule pychecker.checker <script>".

The given implementation also makes the execmodule functionality available from Python code - "from execmodule import execmodule" will give a function 'execmodule' with an interface similar to that for the existing builtin 'execfile'. The difference is that 'execmodule' attempts to locate its argument using the Python module namespace instead of the native filesystem. (This equivalence to execfile is what demands the sys._getframe hack - we want to use the caller's namespace when no globals are provided. For those that dislike sys._getframe, remove this line and make the globals argument mandatory)

Note that, due to a limitation of imp.find_module, 'execmodule' cannot use the system metapath to find modules (e.g. inside zip files). However, files in such locations are unlikely to be usable as scripts, even if they could be located.

(This is a much expanded version of a script originally posted to python-dev by Guido van Rossum. Guido's version was equivalent to the current behaviour of '-m' - it was posted during the discussion on whether or not to implement the command line switch)

8 comments

Thomas Heller 19 years, 5 months ago  # | flag

Works great. If this script is saved as p.py in sys.path somewhere, one could even pretend there were an extended -m switch ;-)

python -mp pychecker.checker mymodule
Paul Moore 19 years, 5 months ago  # | flag

Silly question but... Why isn't this functionality part of the built in support?

Nick Coghlan (author) 19 years, 5 months ago  # | flag

Mainly due to the fact that the top-level module support was easy to build in to the interpreter (basically a single call to _PyImport_FindModule, plus error checking), but you can see for yourself that the full-fledged version is at least moderately complicated (even when written in Python!).

There's an SF patch (#1043356) based on the idea of automatically invoking this Python module from C if the search for a top-level module fails, but the current implementation of the switch has been kept fairly simple. This might have been different if we were further from the first Python 2.4 beta.

Sufficient support/demand on c.l.p. could still result in the interpreter getting native support for executing scripts inside packages. Given how long Python has gone without this feature at _all_, I'm not expecting that to happen before Python 2.5 (if it happens at all)

Nick Coghlan (author) 19 years, 5 months ago  # | flag

The design reason for it. My last comment touched on the practical and political reasons, but there was a design reason too.

The original pitch for the behaviour of the '-m' switch is that it is exactly like executing the located module by specifying its full filename on the command line.

That doesn't hold true for modules inside packages - the containing packages have to be imported in order to reliably locate the modules they contain. The problem is that this doesn't match the behaviour obtained by directly supplying the full path to the script.

Expanding the description of the semantics to cover modules inside packages is certainly possible - but see the previous comment for why that isn't as straightforward as it might first seem.

Bob Ippolito 19 years, 5 months ago  # | flag

pyrun. Well this is what I use as my "pyrun" script.. it seems a bit simpler than what you've got there:

#!/usr/bin/env python
def _run():
    import sys, os, imp
    def imp_find_module(name):
        """same as imp.find_module, but handles dotted names"""
        names = name.split('.')
        path = None
        for name in names:
            result = imp.find_module(name, path)
            path = [result[1]]
        return result
    path = imp_find_module(sys.argv[1])[1]
    sys.argv[:2] = [path]
    sys.path[0] = os.path.dirname(path)
    execfile(path, globals(), globals())
_run()
Nick Coghlan (author) 19 years, 4 months ago  # | flag

The additional complexity in my version is due to the following properties:

  • respect package __path__ variables

  • mimic execfile argument handling

  • mimic the standard '-m' error responses

For my own use, I would probably have written a version more like yours (aside from the __path__ variable issue). However, I'm proposing this as an addition to the standard library for 2.5, so its a bit more polished than most of my personal-use-only scripts.

Nick Coghlan (author) 19 years, 4 months ago  # | flag

In fact, I just realised that the original attempt at allowing dotted names with -m implemented something very similar to this, only in C, rather than Python.

The concept was dropped for Python 2.4 after I realised it didn't actually follow Python's import semantics correctly.

Nick Coghlan (author) 13 years, 8 months ago  # | flag

For the record, this recipe is no longer necessary as of Python 2.5 (the interpreter's own -m switch supports modules inside packages). Python 2.6 allows such modules to use explicit relative imports and 2.7 allows packages with a __main__ submodule to be executed by supplying the package name.

Created by Nick Coghlan on Sat, 9 Oct 2004 (PSF)
Python recipes (4591)
Nick Coghlan's recipes (11)

Required Modules

Other Information and Tasks