Cross-platform Application Development and Distribution

Authors:

Stani (spe.stani.be#gmail.com)

Nadia Alramli (nadiana.com)

Date:

2010/2/19

License:

Creative Commons Share Alike Non Commercial License

This paper is collection of tips, suggestions and lessons we learned from our work on a cross-platform photo batch processor called ‘Phatch’. Python’s cross-platform libraries can perhaps solve 90% of the problems, but the remaining 10% can be tricky to fix. Our paper is aimed at these remaining bits. We are using examples from Phatch throughout the paper in the hope that some of them will save your time and sanity while developing and distributing Python applications. wxPython is used as an example for the GUI, but most of the tips are applicable to other toolkits as well. The main focus of the paper is applications, but most of it is also useful for modules. Cross-platform development is a broad subject that cannot be completely covered in one paper. This is why we try to link to external resources as much as possible. Be sure to check them out.

We presented this paper at PyCon 2010:

Cross-Platform Development

Portable Directory Structure

We suggest arranging your application directory structure first before writing code. This might save you time by minimizing the need to move modules around later. The structure has to be flexible to adapt to different ‘contexts’. The context is defined by:

  1. The platform: Linux, Mac, Windows, ...
  2. The setup: Run from source, package or frozen (py2exe, py2app, ...)

It is important to keep in mind that the file locations of your application will be different depending on the ‘context’, For example:

  • In Linux, Python source code, data files, documentation and translations will live in a different path trees.
  • If you freeze your application with py2app for Mac, it will run from a Mac OS X app bundle. This is a directory structure, wherein for example data files will be located under MyApp.app/Contents/Resources/ and source files compressed in MyApp.app/Contents/Resources/Python/site-packages.zip.

Based on our experience, we recommend you to separate application files into different directories:

  • Data files: icons, images, ...

  • Docs: HTML, restructured text, ...

  • Locale: i18n translations for gettext.

  • Separate directory for each platform where you can put bin files, shortcuts, ...

  • Source code: Python files, further separate:
    • GUI code: wxPython, pyQt, Tkinter, command line, ...
    • Core code: separate the core and GUI code.
    • Library code: useful modules, which are independent from your application and might be used elsewhere.

A simplified directory structure of Phatch:

phatch
 +--data
 +--docs
 +--locale
 +--linux (bin file, '.desktop' file, man page, ..)
 +--mac (bin file, Info.plist, ...)
 +--windows (bin file, '.ico' icon, ...)
 +--source
     +--core
     +--pyWx (graphical user interface)
     +--cli (command line interface)
     +--lib
         +--pyWx
         +--linux
         +--windows
         +--mac

It is important that all images and icons are in the data path and not in the source path. An alternative is to embed images as strings in Python code. wxPython provides the command line tool img2py for this purpose. Note that we use pyWx as the name of the wxPython code directory, and not:

  • gui: in case we change the GUI toolkit in the future or support more than one.
  • wx: this would conflict with the wxPython module wx.

The different contexts make the use of __file__ problematic or even impossible in the case of py2exe. It is expected in the future that distutils will improve handling of data files. We strongly suggest you write one module to handle path locations depending on the context instead of hard-coding locations or repeating path locating logic across your library. Imagine your application has a module named path:

import sys

BIN = False  # should be set to True by /usr/bin/app
SETUP_PY = sys.argv[0].endswith('setup.py')

def platform():
    if sys.platform.startswith('win'):
        return 'windows'
    elif sys.platform.startswith('darwin'):
        return 'mac'
    return 'linux'

def SETUP_PY or BIN:
    return not sys.argv[0].endswith('.py')

def setup():
    if hasattr(sys, 'frozen'):
        return 'frozen'
    elif is_package():
        return 'package'
    return 'source'

CONTEXT = (platform(), setup())

if CONTEXT == ('mac', 'frozen'):
    DATA = os.path.join(APP_ROOT, 'Contents','Resources') # py2app
elif CONTEXT == ('windows', 'frozen'):
    DATA = os.path.join(APP_ROOT, 'data') # py2exe
elif CONTEXT == ('linux', 'package'):
    DATA = os.path.join(sys.prefix, 'share', APP_NAME) # linux
else:
    DATA = os.path.join(...) # when run from source

Afterwards you can import this path module to locate for example the data path:

>>> import path
>>> path.DATA != path.SOURCE
True
>>> icon = os.path.join(path.DATA, 'images', 'icon.png')

As the context doesn’t change during the execution of a program, constant values can be used. For Phatch we used two context modules:

It is also important to add the following files in the root of your package:

  • LICENSE.txt
  • README.txt (best in Restructured Text so PyPi can use it)
  • CHANGES.txt

These files should describe everything necessary for distribution. For example, list dependencies, a description of the installation process, ...

If you are packaging modules instead of an application, be sure to also read the Hitchhiker’s Guide to Packaging.

pythonw

Most Python coders are familiar with python and pythonw. What is relatively unknown, is that they are implemented quite differently on each platform:

  • Windows:

    • python: Start program in an outputting console to display stdout and stderr.
    • pythonw: Start program without an outputting console.
  • Mac OS X: Since python 2.5 the programs python and pythonw link to the same file. Before python2.5 they were really different:

    • python: Run a program without a GUI (Graphical User Interface).
    • pythonw: Run a program with a GUI.

    For example running a wxPython application with python is not possible on Mac OS X with python 2.3 or python 2.4 as wxPython programs have a graphical user interface. (Mac OS X Tiger still shipped with Python 2.3.5 by default. Mac OS X Leopard ships with Python 2.6.)

  • Linux: pythonw does not exist at all!

In order to force a GUI application to use pythonw on Mac OS X, you can restart your application. Example code from phatch/app.py:

import os
import sys

def reexec_with_pythonw(f):
    """'pythonw' needs to be called for any GUI app
    to run from the command line on Mac OS X."""
    if sys.platform == 'darwin' and \
           not sys.executable.endswith('/Python'):
        sys.stderr.write('re-executing using pythonw')
        os.execvp('pythonw', ['pythonw', f]+sys.argv[1:])

Locating

Font Files

Some libraries such as PIL require the full path to font files. Finding fonts on a system in a cross-platform way is not trivial. For example wxPython provides a font dialog which returns font names, but not the font file names. The TTFquery project (http://ttfquery.sourceforge.net) provides a module named findsystem.py, which works very well for Windows. It also provides solutions for Linux and Mac OS X, but those are not optimal as they only search specific directories. For Linux and Mac OS X there are these alternatives:

  • Parse the output of fontconfig, which might be tricky as the format is undocumented and might change in the future.
  • Parse the output of "locate -i .ttf .otf". This is easy to implement and predicatable. For performance reasons it is good to cache the results. This is the method that Phatch uses in phatch/lib/fonts.py.

Executables

  • Finding an executable on Linux and Mac OS X is easy by reading the output of the which command. Simplified example code from phatch/lib/system.py:

    def find_exe(exe):
        return subprocess.Popen('which %s'%exe, stdout=subprocess.PIPE,
            shell=True).stdout.read()
    
    >>> find_exe('python')
    '/usr/bin/python'
  • Finding an executable on Windows is more complicated as there is no easy which command. Here are some possible strategies:

    • Check the source of webbrowser module. It contains some useful code.
    • Execute the commands ftype and assoc to find the appropriate application for a given file extension.
    • Seach the current directory and the PATH while respecting the PATHEXT environment string.
    • Scan the Windows registry for the corresponding file type and retrieve the associated application.
    • A brute force method is to recursively search C:\Program Files for executables (.exe) one or two levels deep.

There is probably a better solution as we are not Windows experts.

Desktop Directory

Every platform has the concept of a ‘Desktop’. Interacting with the desktop enhances the native feel of your application. This is how the desktop directory can be retrieved in a cross-platform way:

  • Mac OS X: The Desktop directory is always in the user home directory.

  • Windows: The Desktop directory can be retrieved from the Windows registry.

  • Linux: Special directory paths (Desktop, Documents, ...) are described in the ~/.config/user-dirs.dirs file if the language is not English. Here is an example file for Dutch:

    # This file is written by xdg-user-dirs-update
    # If you want to change or add directories, just edit the line you're
    # interested in. All local changes will be retained on the next run
    # Format is XDG_xxx_DIR="$HOME/yyy", where yyy is a shell-escaped
    # homedir-relative path, or XDG_xxx_DIR="/yyy", where /yyy is an
    # absolute path. No other format is supported.
    #
    XDG_DESKTOP_DIR="$HOME/Bureaublad"
    XDG_DOWNLOAD_DIR="$HOME/Downloads"
    XDG_TEMPLATES_DIR="$HOME/Sjablonen"
    XDG_PUBLICSHARE_DIR="$HOME/Openbaar"
    XDG_DOCUMENTS_DIR="$HOME/Documenten"
    XDG_MUSIC_DIR="$HOME/Muziek"
    XDG_PICTURES_DIR="$HOME/Afbeeldingen"
    XDG_VIDEOS_DIR="$HOME/Video's"
    

This Python function will retrieve the Desktop directory on all platforms. Example code from phatch/lib/desktop.py:

if sys.platform.startswith('win'): # Windows
    from win32com.shell import shell, shellcon
    DESKTOP_FOLDER = shell.SHGetFolderPath(0,
        shellcon.CSIDL_DESKTOP, None, 0)
elif sys.platform.startswith('darwin'): # Mac OS X
    DESKTOP_FOLDER = os.path.expanduser('~/Desktop')
else: # Linux
    DESKTOP_FOLDER = os.path.expanduser('~/Desktop')
    user_dirs = os.path.expanduser('~/.config/user-dirs.dirs')
    if os.path.exists(user_dirs):
        match = re.search('XDG_DESKTOP_DIR="(.*?)"',
                    open(user_dirs).read())
        if match:
            DESKTOP_FOLDER = os.path.expanduser(
                match.group(1).replace('$HOME', '~'))

Tempfiles

Temporary files are handled quite differently by platforms as described in a blog post from Logilab. Using the mkstemp function of the standard tempfile module requires some caution, as you are left with the responsibility to handle some low level file garbage collection.

The mkstemp function in the tempfile module returns a tuple of 2 values:

  • An OS-level handle to an open file (as it would be returned by os.open()).
  • The absolute pathname of that file.

Often only the absolute pathname is used. This seems to be working fine, but there is a bug lurking there. This bug will show up on Linux if you call this function many times in a long running process. On Windows, it will show up on the first call. The problem is that the file descriptor was not closed. The first element of the tuple returned by mkstemp is typically an integer used to refer to a file by the OS. In Python, Leaving a file open is usually not a big deal because the garbage collector will ultimately close the file for you, but here we are not dealing with file objects, but with OS-level handles. The interpreter sees an integer and has no way of knowing that the integer is connected to a file.

The issue on Windows is that you can’t remove any temporary file without closing the file descriptor, while on Linux you can. So on Windows this bug (removing before closing file descriptor) will show up from the first call of os.remove(temporary_file), while on Linux it only shows up when you exhaust the available file descriptors. Sometimes your application won’t arrive at that point on Linux when it only uses a limited amount of temporary files. In that case your application seems to work fine on Linux, while it doesn’t work on Windows. In reality the code is wrong both for Linux and Windows. That’s another reason why you need to make sure you test your application on every platform.

In Python, this translates to the following code:

from tempfile import mkstemp
fd, temp_path = mkstemp()
subprocess.call('command --output %s' % temp_path)
file = open(temp_path, 'r')
data = file.read()
file.close()
os.close(fd) # do not forget to close the file descriptor
os.remove(temp_path)

For Phatch we wrapped this in class Tempfile in phatch/lib/system.py:

>>> t = TempFile('.png')
>>> t.path.endswith('.png')
True
>>> t.close()

System Thumbnail Cache

Unfortunately each platform has its own way to store thumbnails of files:

Windows

Each directory stores its thumbnails in a hidden thumbs.db file. Unfortunately this file is not straightforward to access with Python in Windows. The vinetto project solves this, but unfortunately it must be run in Linux or Cygwin. (http://vinetto.sourceforge.net/)

Mac OS X

How thumbnails are cached depends on the file system according to this page:

  • HFS:

    The HFS filesystem, for reasons of compatibility with ancient Apple operating systems, stores files in two “forks”: a data fork, which is what you normally think of as a file, and a resource fork, which is an indexed set of typed data objects. For those of us accustomed to the concept of single-forked files, the resource fork is sort of mysterious. It is part of the file, but not part of the byte stream you would receive if you read the file. In the old days the resource fork of an application would store things like strings, user interface elements, icons and bitmaps, and other blocks of largely immutable data. These days it’s hardly used at all except by Finder.

    You could reasonably conceive a file’s resource fork as being filesystem metadata, but from a practical perspective that’s not actually the case. HFS has metadata support, but it’s distinct from the concept of forked files. Other filesystems (like NTFS) supports multiple streams of data addressed by the same filename, but such capabilities are almost entirely unused.

    In any case, the technical answer to “where is the file’s resource fork?” is “in the file’s resource fork.” Where is it stored? In the file’s resource fork.

  • FAT:

    FAT-16 or FAT-32, which has no support for resource forks. These are put in dot-files (files with the same name, but with a . in front of them), which the Finder ignores. You can see these from the Terminal. Windows ignores them too because they usually have the -h (hidden) attribute.

    Mac OS X does this on all filesystems that don’t support resource forks.

Linux

Thumbnails are cached in the ~/.thumbnails directory according to the Freedesktop specification (http://jens.triq.net/thumbnail-spec/). This standard is valid for all desktop environments such as Gnome and KDE. Most open source applications which support thumbnails on multiple platforms (such as Gimp and Blender) use the Freedesktop standard for all platforms, as it is straightforward to implement. For Phatch we developed a library in pure python, which enables to use the Freedesktop standard on all platforms for thumbnails: phatch/lib/thumbnail.py.

Platform Specific Tools or Libraries for Desktop Integration

Windows

Linux

The most widely used free software X desktops, GNOME, KDE and Xfce, are working closely with the Freedesktop.org project. This organization seeks to ensure that differences in development frameworks are not user-visible. Freedesktop.org was formerly known as the X Desktop Group, and the acronym “XDG” remains common in their work:

  • PyXDG is a python library to access Freedesktop.org standards. Currently supported are:

    • Base Directory Specification.
    • Menu Specification.
    • Desktop Entry Specification.
    • Icon Theme Specification.
    • Recent File Specification.
    • Shared-MIME-Database Specification.

    These are three pyxdg examples:

    1. Before the application data was stored in a hidden ~/.application directory. This method is now deprecated and three different directories have to be used for configuration, data and cache files:

      >>> import xdg.BaseDirectory as bd
      >>> bd.xdg_config_home # preferences/settings/...
      '/home/user/.config'
      >>> bd.xdg_data_home # user plugins/images/...
      '/home/user/.local/share'
      >>> bd.xdg_cache_home # user cached items
      '/home/user/.cache'
      
    2. You can also view and edit the list of the recently-opened files:

      >>> import xdg.RecentFiles as rc
      >>> all = rc.RecentFiles()
      >>> all.parse()
      >>> last = all.getFiles()[0]
      >>> last.URI
      u'file:///home/user/recent_file.txt'
      
    3. The Python standard library has a module called “mimetypes”, but it works using lists of mapped file extensions. The xdg.Mime module allows to determine file mime type very accurately using magic number tests. The advantage is that if a file has wrong or no extension, the test will return the correct MIME:

      >>> import xdg.Mime
      >>> xdg.Mime.get_type('icon', name_pri=0)
      'image/png'
      

    For more information visit: http://www.Freedesktop.org/wiki/Software/pyxdg

  • xdg-utils http://portland.Freedesktop.org/wiki/

    • Xdg-utils consists of the following installation related tools:

      • xdg-desktop-menu, install desktop menu items.
      • xdg-desktop-icon, install icons to the desktop.
      • xdg-icon-resource, install icon resources.
      • xdg-mime, query information about file type handling and add descriptions for new file types.
    • It also contains the following runtime integration tools:

      • xdg-open, open a file or URL in the user’s preferred application.
      • xdg-email, send mail using the user’s preferred e-mail composer.
      • xdg-screensaver, control the screensaver.

Processing Filenames

Pitfalls

Be sure your application can handle filenames that contain:

  • Non-ASCII characters: A rule of thumb is to use Unicode for paths you pass to functions like os.listdir. That will cause Python to call the Unicode version of the function, and you get the true Unicode filenames. For examples and more details refer to this article.

  • Spaces: To be safe it is better to preprocess filenames and add quotes only if necessary. Example code from phatch/lib/system.py:

    import re
    
    RE_NEED_QUOTES = re.compile('^[^\'"].+?\s.+?[^\'"]$')
    
    def fix_quotes(text):
        """Fix quotes for a command line parameter. Only surround
        by quotes if a space is present in the filename.
        """
        if not RE_NEED_QUOTES.match(text):
            return text
        if not ('"' in text):
            return '"%s"'%text
        elif not ("'" in text):
            return "'%s'"%text
        else:
            return '"%s"'%text.replace('"', r'\"')
    

    Demonstration:

    >>> fix_quotes('blender')
    'blender'
    >>> fix_quotes('/my programs/blender')
    '"/my programs/blender"'
    

Format

Filenames on Linux can be formatted differently:

  • There are two common formats:

    • Normal filename notation: /home/user/foo.png.
    • URI notation: file:///home/user/foo.png.
  • Not all Linux file browsers use the same format, when a file location is copied to the clipboard or when a file is dragged & dropped:

    • Nautilus: /home/user/foo.png
    • Thunar: file:///home/user/foo.png

    This can be solved easily with the following function:

    import urllib
    
    def fix_path(path):
        if path.startswith('file://'):
            return urllib.unquote(path[7:])
        return path
    
    >>> fix_path('file://hello%20world')
    'hello world'

Python, Batteries Included

There is a handful of useful methods and modules to use when developing cross-platform applications. The os.path module offers path manipulation functions where you don’t have to worry about platform differences. The platform module gives you access to underlying platform’s identifying data. It is very useful when you need to write a platform specific piece of code without worrying about other platforms.

  • os

    • os.path.join(path1 [, path2 [, ...]]): Join one or more path components intelligently.

    • os.path.split(path), os.path.splitdrive(path) and os.path.splitext(path): Convenient path splitting functions.

    • os.linesep: The string used to separate (or, rather, terminate) lines on the current platform.

    • os.pardir: The constant string used by the operating system to refer to the parent directory.

    • os.environ: This can be used to determine which desktop environment is running in Linux:

      'GNOME_DESKTOP_SESSION_ID' in os.environ
      'KDE_FULL_SESSION' in os.environ
      
  • sys

    • sys.platform: Platform identifier, e.g., 'win32', 'darwin', 'linux2', etc.
    • sys.prefix: A string giving the site-specific directory prefix where the platform independent Python files are installed.
  • platform

    • platform.machine(): Returns the machine type, e.g., 'i386'
    • platform.architecture(executable=sys.executable): Returns information about the bit architecture of a given executable.

If you use shell=True with the subprocess module, pass the command line as a string, not as an argument list. On Linux it will fail, when the arguments contain spaces or only a command line option:

.. sourcecode:: python

    >>> from subprocess import Popen
    >>> Popen(['gcc', '--version'], shell=True)
    gcc: no input files

On Windows an argument list works correctly:

>>> from subprocess import Popen
>>> Popen(['gcc', '--version'], shell=True)
gcc (GCC) 3.4.5 (mingw-vista special r3) ...

Opening Files with the Default Application

The os module does not provide a cross-platform way to start a file with the associated application. Luckily there is any easy way to do it. An example from phatch/lib/system.py:

def start(path):
    if hasattr(os, 'startfile'): # Windows
        os.startfile(path)
    else:
        if sys.platform.startswith('darwin'): # Mac OS X
            command = 'open'
        else: # Linux
            command = 'xdg-open'
        subprocess.call([command, path])

Creating Shortcuts

Phatch needed to create shortcuts dynamically. For example a desktop shortcut on which images could be dropped.

Windows

Shortcuts in Windows are defined by .lnk files. These files can not be created with file write. However you can create them using the pywin32 API.

Phatch uses the following code to generate Windows shortcuts dynamically in phatch/lib/windows/shortcut.py:

import win32com.client

def create(save_as, path,arguments = "", working_dir = "",
        description = "", icon_path = None, icon_index = 0):
    # initialize shortcut
    shell = win32com.client.Dispatch("WScript.Shell")
    shortcut = shell.CreateShortCut(save_as)

    # set shortcut parameters
    shortcut.Targetpath = path
    shortcut.Arguments = arguments
    shortcut.WorkingDirectory = working_dir
    shortcut.Description = description
    if icon_path:
        shortcut.IconLocation = icon_path

    # save shortcut
    shortcut.save()

Linux

Shortcuts in Linux are defined in .desktop files. These files provide information about your application such as the name, icon, and description. You can find the specifications here. They are used for application launchers and to create menus. The .desktop files should be made executable, unless they are placed in /usr/share/applications (see Start Menu topic).

Phatch uses the following code to generate a Linux .desktop shortcut dynamically in phatch/lib/linux/desktop.py:

import os

DROPLET = """\
#!/usr/bin/env xdg-open
[Desktop Entry]
Version=1.0
Type=Application
Name=%(name)s
Terminal=false
Exec=%(command)s
Icon=%(icon)s"""

def create_droplet(name, command, folder='~/Desktop', icon='gnome-panel-launcher'):
    filename = os.path.expanduser(os.path.join(folder, name + '.desktop'))
    data = {'name': name, 'icon': icon, 'command': command}
    droplet = open(filename, 'w')
    droplet.write(DROPLET % data)
    droplet.close()
    os.chmod(filename, 0755) # make it executable

Here is an example output:

#!/usr/bin/env xdg-open
[Desktop Entry]
Version=1.0
Type=Application
Name=Image Inspector
Terminal=false
Exec=phatch -n %U
Icon=phatch

Note the %U beside the exec command name. This is a special field code that will be expanded by the file manager or program launcher when encountered in the command line. Field codes consist of the percentage character (“%”) followed by an alpha character. In this case %U will be expanded into a list of URLs which is very important if you want the shortcut to act as a multi-file droplet. The .desktop specification website has the full list of special field codes and description.

Mac OS X

The concept of ‘shortcuts’ as in Windows and Linux does not exist on Mac OS X. The most similar concept is aliases.

Adding Your Application to the Start Menu

Linux

A package can include a .desktop file. Your setup script needs to copy this desktop file to /usr/share/applications. Otherwise, it won’t appear in the start menu. Set the Categories field to the name of the menu. Here is for example the Phatch shortcut:

[Desktop Entry]
Version=1.0
Exec=phatch %U
Icon=phatch
Terminal=false
Type=Application
Categories=Graphics;Photography;GTK;

Note

Desktop shortcuts in /usr/share/applications should not be executable with #!/usr/bin/env xdg-open.

Windows

Adding to the start menu is a matter of creating a shortcut in the right location:

  • If you are using distutils’ bdist_wininst command to generate a Windows installer, then you can use the create_shortcut function in the post-install script:

    create_shortcut(target, description, filename[, arguments[, workdir[, iconpath[, iconindex]]]])
  • You can create shortcuts programatically using pywin32 as demonstrated before. The only difference is that the shortcut has to be placed in the start menu folder:

    from win32com.shell import shell, shellcon
    start_menu_folder = shell.SHGetSpecialFolderPath(0, shellcon.CSIDL_COMMON_STARTMENU)
    
  • Another option is to use a Windows installer, like Inno Setup or NSIS, to create the shortcut.

Associating Actions with File Types

You can enhance the interaction with your program by registering special actions with file types. These actions will appear in the context menu of the file browser. If appropriate, implement the Open action as it will allow the user to double click a file to open your application. This is easily achieved by registering file types with your application. Other actions can be added in different ways depending on the platform or file browser. Do not exaggerate and clutter the user experience.

Linux

The Open action is defined by the MimeType fields of /usr/share/applications/*.desktop files. This works for any application, not just file browsers. If you develop a JPEG application, it will be automatically integrated in photo browsers, such as F-Spot. This is a simplified example extracted from the Phatch’s desktop file:

#!/usr/bin/env xdg-open
[Desktop Entry]
...
MimeType=application/x-phatch;image/gif;image/jpeg;image/jpg;image/png;image/tiff;inode/directory;

Defining this field will associate your application with the MIME types of your choice. Note that inode/directory is the MIME type for directories.

For additional actions in the context menu, you need to write configurations specific to the file browser:

  • Nautilus:

    • If the package python-nautilus is installed, actions can be defined by a Python file in ~/.nautilus/python-extensions. This requires the dependency python-nautilus.
    • For Phatch we wrote a library to create Nautilus actions dynamically through Python: phatch/lib/linux/nautilusExtension.py
  • Thunar file manager:

    • Actions can be defined in the XML config file ~/.config/Thunar/uca.xml.
    • For Phatch we wrote a library to create Thunar actions dynamically through Python: phatch/linux/thunar.py.

Mac OS X

File type association is typically done by including a property list file with a .plist (e.g. Info.plist) extension in your application bundle. To define file types supported by application you need to fill in the CFBundleDocumentTypes property, for example:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC -//Apple Computer//DTD PLIST 1.0//EN http://www.apple.com/DTDs/PropertyList-1.0.dtd >
<plist version="1.0">
...
<key>CFBundleDocumentTypes</key>
<array>
    <dict>
        <key>CFBundleTypeExtensions</key>
        <array>
            <string>html</string>
            <string>htm</string>
        </array>
        <key>CFBundleTypeIconFile</key>
        <string>icon.icns</string>
        <key>CFBundleTypeName</key>
        <string>HTML Document</string>
        <key>CFBundleTypeRole</key>
        <string>Viewer</string>
    </dict>
</array>
</plist>

You can view/edit .plist files using the “Property List Editor” application available as part of Apple Developer Tools.

The plist file should be located in the Contents directory of the application bundle, for example:

MyApp.app/Contents/Info.plist

Windows

You can register file types and add actions to the context menu by writing to the Windows registry. This is typically done by an installer program such as Inno Setup. You can also do it at run time using Python. We created a library to do this for Phatch, you can find it in phatch/lib/windows/register.py. Here is a simplified example:

def register(label, action, filetype='Python.File', suffix='"%1"'):
    try:
        k = '%s\\shell\\%s' % (filetype, label)
        key = _winreg.CreateKey(_winreg.HKEY_CLASSES_ROOT, k)
    except:
        pass
    try:
        command = '%s %s'%(action, suffix)
        _winreg.SetValue(key,"command", _winreg.REG_SZ,command)
        return key
    except:
        return False

Alternatively you can use assoc and ftype. They write to the registry on your behalf:

  • assoc associates a file extension with a file type.
  • ftype associates a file type with a program.

For example:

C:\temp>assoc .pdf
.pdf=AcroExch.Document

C:\temp>ftype AcroExch.Document
AcroExch.Document="C:\Program Files\Adobe\Reader 8.0\Reader\AcroRd32.exe" "%1"

Cross-Platform GUI Development

Human Interface Guidelines

Creating cross-platform applications is not just a technical issue, but also a design issue. Each serious platform has Human Interface Guidelines (HIG) that define design rules. Unfortunately they are all different. They are partly based on usability studies, but also contain arbitrary conventions.

Sensitive User Interface Differences

Each platform has its own differences and variations when it comes to user experience.

Document Interface

  • There are three types of a document interfaces:

  • The use varies between platforms and applications:

    • Mac OS X: SDI, TDI (mostly for web browsers or certain text editors)
    • Linux: TDI, SDI
    • Windows: MDI, TDI, SDI
  • It would take quite some effort to satisfy everyone and support all three interfaces. In case you select one, choose a single or tabbed document interface depending on the type of application, as the Multiple Document Interface is only supported by Windows. A good compromise is TDI where tabs can be detached to a new frame like in Firefox or Google Chrome.

Different Widget Behavior

Take into account that user interface widgets might behave differently on each platform. For example the native notebook widget in wxPython:

  • Not all platforms support vertical tabs.
  • Not all platforms support multiple row of tabs.
  • In Aqua notebooks on Mac OS X, all tabs have to be visible and can’t hide behind arrow buttons.

Linux Desktop Environments

If you target Linux be sure to test your application both in a Gnome and KDE as the difference in rendering might cause some layout bugs.

System Colors

Mac OS X and Windows have a fixed color palette, however Linux users feel strongly about their color palettes. For example, most users change Ubuntu’s default brown palette.

  • Use system colors whenever possible. Try to avoid:

    • Introducing your own color palette.
    • Using a system color from one platform (for example: the background color), as a default color on another platform.
  • Cross-platform GUI kits normally give you API access to the system colors. For example in wxPython:

    >>> import wx
    >>> [color for color in dir(wx) if color.startswith('SYS_COLOUR')]
    ['SYS_COLOUR_3DDKSHADOW', 'SYS_COLOUR_3DFACE', 'SYS_COLOUR_3DHIGHLIGHT',
     'SYS_COLOUR_3DHILIGHT', 'SYS_COLOUR_3DLIGHT', 'SYS_COLOUR_3DSHADOW',
     'SYS_COLOUR_ACTIVEBORDER', ....]
    
  • For Phatch we developed code in wxPython to draw a nice gradient based on Linux system colors in phatch/lib/pyWx/vlist.py:

    def GradientColour(color):
        rgb = r, g, b = color.Red(), color.Green(), color.Blue() # wx2.6
        m = max(rgb)
        rgb_without_max = [x for x in rgb if x != m]
        if not rgb_without_max:
            return wx.Colour(128, 128, 128)
        n = max(rgb_without_max)
        keyw = {}
        for c in ('Red', 'Green', 'Blue'):
            x = getattr(color, c)()
            if x == m:
                keyw[c.lower()] = x
            elif x == n:
                keyw[c.lower()] = x/2
            else:
                keyw[c.lower()] = x/8
        return wx.Colour(**keyw)
    

    This is used as follows for the add actions dialog:

    hilight = wx.SystemSettings_GetColour(wx.SYS_COLOUR_MENUHILIGHT)
    color_from = GradientColour(hilight)
    color_to = hilight
    

Icons

Stock Icons

  • If you target Linux, you need to be familiar with the notion of ‘stock icons’. Stock icons are provided by the user theme. They ensure consistent artwork throughout different applications. Use stock icons wherever possible and try to avoid custom icons.

  • Most frequently used stock icons are accessible through the wxPython API:

    >>> import wx
    >>> [icon for icon in dir(wx) if icon.startswith('ART_')]
    ['ART_ADD_BOOKMARK', 'ART_BUTTON', 'ART_CDROM', 'ART_CMN_DIALOG', 'ART_COPY',
     'ART_CROSS_MARK', 'ART_CUT', 'ART_DELETE', 'ART_DEL_BOOKMARK', 'ART_ERROR',
     'ART_EXECUTABLE_FILE', 'ART_FILE_OPEN', 'ART_FILE_SAVE', ...]
    >>> file_open_bitmap = wx.ArtProvider_GetBitmap(wx.ART_FILE_OPEN, wx.ART_OTHER, (48, 48))
    
  • This is the equivalent code for the GTK API:

    >>> import gtk
    >>> [i for i in dir(gtk) if i.startswith('STOCK_')]
    ['STOCK_ABOUT', 'STOCK_ADD', 'STOCK_APPLY', 'STOCK_BOLD', 'STOCK_CANCEL',
     'STOCK_CAPS_LOCK_WARNING', 'STOCK_CDROM', 'STOCK_CLEAR', 'STOCK_CLOSE',
     'STOCK_COLOR_PICKER', 'STOCK_CONNECT', 'STOCK_CONVERT', ...]
    
  • Use pyXDG to access other icons:

    >>> import xdg.IconTheme
    >>> xdg.IconTheme.getIconPath('firefox')
    '/usr/share/pixmaps/firefox.png'
    >>> xdg.IconTheme.getIconPath('fileopen', size=48, theme='gnome')
    '/usr/share/icons/gnome/scalable/actions/fileopen.svg'
    
  • Choose your own set of icons for Windows and Mac OS X as they don’t provide (freely-licensed) stock icons.

  • Use the standardized icon naming specification of Linux to make theme swapping easy.

License

  • If you use icons from the web, make sure to check license compatibility.

  • Try to avoid mixing licenses unless you know what you are doing.

  • Debian/Ubuntu only accepts version 3 of Creative Commons Share Alike to combine with GPL or LGPL. The simplest solution is often to contact the author and ask:

    • To upgrade his CC-SA license to version 3.
    • To dual license the icons.

Avoid Crippled Looking Icons

  • If icon sizes are known beforehand, pre-render them in the correct size and make sure they are nicely anti-aliased.
  • Make sure that toolbar icons are all exactly the same size.
  • Use PNG instead of GIF for transparent icons.

Resources

  • Linux: gnome-look.org, kde-look.org, …

    • Tango is a ‘neutral theme’ and is designed to work also well on Windows and Mac OS X.
    • Humanity: orange-based, default theme of Ubuntu.
  • Open Clipart Library: openclipart.org:

    • SVG icons are scalable and resolution independent. These can be rendered off-line to the appropriate format for your platform.
    • The License is ‘public domain’ which also allows commercial use.
  • Other:

Notifications

There are occasions in which your application wants to notify the user about something: software updates are available, a new instant message have been received, the 300 page print job has finally finished, etc.

  • To keep notifications easy to port cross-platform, don’t make them interactive. For example Ubuntu does not support notifications that require user interaction.

  • These are the most important libraries:

    • Linux: pynotify.
    • Mac OS X: Growl, which is not standard, is usually installed.
    • Windows: a good wxPython solution is ToasterBox of Andrea Gavana, which mimics the look of Firefox or Thunderbird notifications.
  • For Phatch we developed a library that unifies these three systems in one API: phatch/lib/notify.py.

Events

Model–View–Controller (MVC) is an architectural pattern which is often recommended for GUI programming. If this pattern seems overkill for your application, try at least to separate the GUI code (View) clearly from the other code. Let the two code parts interact with each other through events.

  • First, it is good to develop a simple command line application with the most basic functionality of your application. Afterwards build the GUI part on top.

  • PyPi lists many event frameworks e.g. pubsub

    • http://pubsub.sourceforge.net.

    • Hello world example:

      '''
      One listener is subscribed to a topic called 'rootTopic'.
      One 'rootTopic' message gets sent.
      '''
      from pubsub import pub
      
      # create a listener
      def listener1(arg1, arg2=None):
          print 'Function listener1 received:\n  arg1="%s"\n  arg2="%s"' % (arg1, arg2)
      
      # register listener
      pub.subscribe(listener1, 'rootTopic')
      
      # create a function that sends a message
      def doSomethingAndSendResult():
          print 'lahdida... have result, publishing it via pubsub'
          pub.sendMessage('rootTopic', arg1=123, arg2=dict(a='abc', b='def'))
      
      # define the main part of application
      if __name__ == '__main__':
          doSomethingAndSendResult()
      

      This will produce the following output:

      lahdida... have result, publishing it via pubsub
      Function listener1 received:
        arg1="123"
        arg2="{'a': 'abc', 'b': 'def'}"

wxPython

wxPython is a GUI toolkit for the Python programming language. It is implemented as a Python extension module that wraps the popular wxWidgets cross-platform GUI library. Currently-supported platforms are 32-bit Microsoft Windows, most Unix or Unix-like systems (wrapping GTK), and Mac OS X. Other examples of cross-platform GUI toolkits are pyQt and Tkinter.

Resources

The best resources to learn wxPython are:

Clipboard

Supporting interaction with clipboard greatly enhances the integration with the desktop and is easy to implement:

  • Example code to read the clipboard:

    if not wx.TheClipboard.IsOpened():  # may crash, otherwise
        clip_data = wx.TextDataObject()
        wx.TheClipboard.Open()
        success = wx.TheClipboard.GetData(clip_data)
        wx.TheClipboard.Close()
        if success:
            self.text.SetValue(clip_data.GetText())
        else:
            self.text.SetValue("There is no data in the clipboard in the required format")
    
  • Example code to write to the clipboard:

    clip_data = wx.TextDataObject()
    clip_data.SetText("Hi folks!")
    wx.TheClipboard.Open()
    wx.TheClipboard.SetData(clip_data)
    wx.TheClipboard.Close()
    
  • More info at http://wiki.wxpython.org/ClipBoard

Layout

  • Use sizers instead of fixed coordinates for the placement of widgets. Sizers will adapt the layout of the widgets dynamically when the frame is resized. The same user interface has probably different pixel sizes on different platforms, so relying on absolute pixel coordinates is a bad idea.
  • Leave enough border space around controls, such as buttons.
  • If the design of your user interface is static, use a GUI builder tool such as wxGlade or XRCed. It allows you to layout faster and the code it generates has most of its bugs ironed out due to massive testing.
  • If you use a split panel with a user draggable sash, ensure a minimum width value for each side. Otherwise child controls might get cropped.
  • Try not to change font sizes of buttons. Otherwise text might be cropped.

Debugging

  • Change the order of the main controls. For example if the toolbar is initialized at the wrong moment, your application might crash.
  • Be careful with focus events as they might be triggered after the window has been destroyed. For more details check the wxPython wiki.

Sensitive Issues with Mac OS X

  • To display a context menu, use a wx.ContextMenuEvent, not mouse down or up events. An Apple mouse has only one button. Right-click and Ctrl-Left-click are both ContextMenuEvents on Mac OS X. As gesture user interface develops, other gestures may generate a ContextMenuEvent.

  • Make sure that widgets which can have multiple subitems such as a listbox or a drop down control, are not empty when the GUI frame is initialized. Instead prepopulate them with some dummy value(s) such as Select which can be replaced later. Otherwise your application might crash on Mac OS X.

  • wxPython defines -1 as the default initial value for indices, sizes, position, ... Avoid explicitly writing the default value of -1 in your code.

    • It can crash applications on Mac OS X.
    • wx.DefaultSize (-1, -1) which is always transformed to (0, 0) on Mac OS X. Use reasonable minimal values instead, such as wx.Size(16,16).
  • Turn on anti-aliasing in the STC control:

    SetUseAntiAliasing(True)
    
  • wxPython provides these overwritable wx.App methods to deal with Apple events:

    • MacNewFile () Called in response of an “open-application” Apple event.
    • MacOpenFile (filename) Called in response of an “open-document” Apple event.
    • MacOpenURL (url) Called in response of a “get-url” Apple event.
    • MacPrintFile (fileName) Called in response of a “print-document” Apple event.
    • MacReopenApp () Called in response of a “reopen-application” Apple event.

Miscellaneous

  • Always place a wx.Panel in a wx.Frame, to avoid rendering problems on Windows.
  • If you develop custom widgets, use Cairo instead of the raw wxPython DC blit functions. It gives nicer-looking results and the code is more portable.
  • For more information about cross-platform wxPython recipes see: http://wiki.wxpython.org/RecipesCrossPlatform

Known Limitations

  • Progress dialogs can never be resizable. So be sure to display text with the maximum needed length at the beginning, otherwise longer texts might be cropped.
  • Unfortunately wxPython does not allow dragging & dropping notebook tabs with the native wx.NotebookCtrl. As a workaround, you can choose a non-native notebook ctrl such as FlatNotebook, which looks similar to the tabs in Google Chrome.
  • There is no stable HTML rendering widget. However, wxWebkit is already available as a beta.
  • wxPython does not run on mobile phones.

Web GUI

As more and more applications are available on the cloud, it might be worth considering building your application as a web application instead of a desktop application.

Distribute as Desktop Application

Mozilla Prism and Google Chrome allow you to distribute a web page as a desktop application.

System Interaction

  • Cloud applications lack certain aspects of necessary interaction with the operating system:

    • Dragging & dropping multiple files from a local computer to a web page is not yet supported.
    • For some issues there are some workarounds, which usually requires Flash. For example to upload multiple files (http://www.uploadify.com/).
    • HTML5 will address many of these limitations.
    • One possible strategy for now is to use a GUI toolkit only for the frame and embed a Gecko or Webkit control to display a web page. The wxPython, GTK or pyQt frame can handle all the missing functionality such as drag and drop. Gwibber, an open source microblogging client for GNOME, is constructed this way (https://launchpad.net/gwibber).

Interesting Projects

  • Pyamas: Like the Google Web Toolkit (Java), pyjamas compiles an application (and its libraries, UI widgets and DOM classes) from Python to Javascript. Pyjamas handles the packaging of the generated JavaScript and takes care of browser interoperability issues. This leaves you free to focus on application development. (http://pyjs.org/)
  • Raphaël is a small JavaScript library that will simplify your work with vector graphics on the web. For example: if you want to create your own chart or an image widget which can crop and rotate, this can be simply and easily developed with this library. Raphaël could be considered the web alternative to Cairo, which is used by GTK to draw custom controls.(http://raphaeljs.com)

Disadvantages

  • At some point it will require knowledge of JavaScript and server architecture. If you are not familiar with these, maybe it is not worth the hassle for developing desktop applications.
  • The problem of platform compatibility between Linux, Windows and Mac OS X, is deferred to the compatibility problems between Firefox, Google Chrome, Safari, ... and (especially older versions of) Internet Explorer.

Application Distribution

Scope

We will focus on distributing your application to end users who are probably not familiar with Python. Therefore we will limit ourselves to:

  • Freezing your application for Windows and Mac OS X.
  • Submitting your application to Linux repositories.

If you want to distribute python tools or modules, you should have a look at PyPi, but this is out of the scope of our study. Freezing for Linux is also possible, but this is not the common approach.

We will not fully cover setup.py, py2app or py2exe. Instead we will give some less obvious and hopefully useful tips. For more information about them please refer to the official documentation.

Freezing

Windows: py2exe

py2exe is a distutils extension to convert Python scripts into executable Windows programs. The generated executable can run without a Python installation.

Usage Example

In setup.py write the following:

from distutils.core import setup
import py2exe # loading py2exe so that it can add its command.

setup(
    windows=[
        {
            'script': 'MyApp.py',
            'icon_resources': [(1, 'resources/icon.ico')],
        }
    ]
)

Then simply run the script like this:

python setup.py py2exe

Of course in real applications things are not that easy...

Under the Hood
  • py2exe uses Python’s modulefinder to examine your script and find all Python and extension modules required to run it.
  • Pure Python modules are compiled into .pyc or .pyo files in a temporary directory.
  • A zip archive is built, containing all python files from this directory.
  • Your main script is inserted as a resource into a custom embedded python interpreter supplied with py2exe, and the zip archive is installed as the only item on sys.path.

For more details visit http://www.py2exe.org/old/.

Locating Your Executable Path

__file__ is not available in the frozen main script, That’s one of the reasons we suggest using sys.executable to get your application executable path:

if hasattr(sys, 'frozen'):
    return os.path.dirname(unicode(sys.executable,
        sys.getfilesystemencoding()))

Note that sys.executable uses the file system encoding.

Missing DLLs
  • The Python interpreter was compiled using Microsoft Visual C, so your new program needs the Microsoft Visual C runtime DLL to run.
  • If you are using Python 2.4 or 2.5, the DLL you need is called MSVCR71.dll. It is included in the Microsoft Visual C++ 2005 Redistributable Package.
  • For Python 2.6, the DLL you need is called MSVCR90.dll. You can find it in Microsoft Visual C++ 2008 Redistributable Package.
  • If you have the right to redistribute the DLL, include it with the manifest file as a data file in the setup options.
  • If you don’t have the right to redistribute it, you can include the Microsoft Visual C++ Redistributable Package in your application installer. The package is redistributable by anyone.
  • When faced with errors that don’t make sense, delete the build directory. This may help with obscure DLL-Not-Found-Problems when updating wxPython.

The full instructions about providing the Microsoft Visual C runtime DLL are in the py2exe tutorial

Distributing Your Application

There are many options to build Windows installers, including:

  • Inno Setup: Free software.
  • NSIS: Open source.

Mac OS X: py2app

py2app is a Python setuptools command that allows you to make standalone Mac OS X application bundles from Python scripts. It is similar in purpose and design to py2exe.

Usage Example

In setup.py write the following:

from setuptools import setup
setup(
    setup_requires=["py2app"],
    app=["MyApp.py"],
    options={
        'py2app': {
            'iconfile': 'resources/icon.icns',
        }
    }
)

Then simply run the script:

python setup.py py2app

Of course, things can be a little more complicated in a real application.

Under the Hood
  • An application bundle will be created with the name of your application, for example MyApp.app. This is the root directory of your application bundle.
  • The Contents/Info.plist file will be created from the dict or filename given in the plist option. py2app will fill in missing keys as necessary.
  • The main script of your application will be copied as-is to the Contents/Resources/ directory of your application bundle.
  • Packages that were explicitly included with the packages option, or by a recipe, will be placed in Contents/Resources/lib/python2.X/.
  • A zip file containing all Python dependencies including your application source code is created at Contents/Resources/lib/Python2.X/site-packages.zip.
  • Extensions (which can’t be included in the zip) are copied to Contents/Resources/lib/python2.X/lib-dynload/

For more details check the official py2app documentation.

Making Your Application a Drop Target for Files (Droplet)

Droplets are scripts that can handle files dropped into its icon. To make your application a drop target for files, you can do the following:

  • Override wx.App.MacOpenFile(self, filename) with code for loading the file. For example:

    class MyApp(wx.App):
        def OnInit(self):
            #do init stuff here
        def MacOpenFile(self, filename):
            print filename #code to load filename goes here.
  • Set the argv_emulation option in py2app to True. This will put the names of the dropped files into sys.argv. In fact you should set this option to True if your application accepts any kind of command line arguments.

  • Associate file types with your application as explained previously.

For more details and tips check: wxPython Optimizing for Mac OS X

Customizing Application Properties

py2app allows you to make some tweaks to your Info.plist file to change how your application behaves and interacts with Mac OS X. A property list file will be created from the dict given in the plist option and py2app will fill in any missing keys as necessary. File type association explained in a previous section can be done like this with py2app:

# Associating html and htm files with your application
Plist = {
    'CFBundleDocumentTypes': [{
        'CFBundleTypeExtensions': ['html','htm'],
        'CFBundleTypeName': 'HTML Document',
        'CFBundleTypeRole': 'Viewer',
        'CFBundleTypeIconFile': 'Icon.icns',
    }]
}
setup(
    app=['MyApp.py'],
    options={
        'py2app': {'plist': Plist}
    },
)

Here are some commonly customized property list keys relevant to py2app applications:

CFBundleGetInfoString:
 The text shown by Finder’s Get Info panel.
CFBundleIdentifier:
 The identifier string for your application (in reverse-domain syntax), e.g. "org.pythonmac.py2app".
CFBundleURLTypes:
 An array of dictionaries describing URL schemes supported by the bundle.
LSBackgroundOnly:
 If True, the bundle will be a faceless background application.
LSUIElement:If True, the bundle will be an agent application. It will not appear in the Dock or Force Quit window, but still can come to the foreground and present a UI.
NSServices:An array of dictionaries specifying the services provided by the application.
Distributing Your Application
  • Apple Disk Image (.dmg) is usually the simplest way to distribute Mac OS X applications. There is a command line utility hdiutil to create the image:

    hdiutil create MyApp.dmg -srcfolder /path/to/app/
  • For a fancier installer, create an installation package *.pkg. There is a PackageMaker application as part of Apple Developer Tools.

Common Issues with Frozen Applications

  • Using __import__ or otherwise importing code without the usage of the import statement.
    • Use explicit imports when possible
    • You can include undetected dependencies using the includes option
  • Requiring in-package data files
    • Separating data files and source code is a good practice
    • You can disable compression in py2exe by setting the skip_archive option to True this will only help if you want to use path functions to locate python modules.
  • Always delete the generated dist directory before every new build. Otherwise you may get confusing errors.

GUI2EXE

GUI2EXE is a Graphical User Interface frontend to all the “executable builders” available for the Python programming language. It can be used to build standalone Windows executables, Linux applications and Mac OS X application bundles and plugins starting from Python scripts.

At the moment the supported executable builders are:

  • py2exe
  • py2app
  • PyInstaller
  • cx_Freeze
  • bbFreeze
  • vendorID

GUI2EXE has some extra advanced features for py2exe only, such as:

  • Automatic generation of simple Inno Setup scripts.
  • Possibility to use UPX compression on dlls/exes while compiling.
  • Show a list of modules py2exe thinks are missing or a list of binary dependencies (dlls) py2exe has found.

The website has an extensive list of features. The author of GUI2EXE, Andrea Gavana, is very responsive on the wxpython-users mailing list in case you run into any issues.

Linux

Good Practices

No matter which distribution you are targeting, we suggest you contact the distribution maintainers to do the packaging. But first make their job easier by following some good practices during development:

  • Clearly state the license of the code and make sure that there is a copy of the license in the tarball.

  • Document the building process as much as possible:

    • Building procedure
    • Dependencies
    • Data files location
  • Separate data files from source files. This is important for Windows and Mac applications as well.

  • Avoid using __file__ to locate files:

    • __file__ is not always defined, py2exe is an example.
    • The Python compiled files .pyc can live in different directories. Using __file__ might cause issues in some cases.
  • Don’t ship pre-compiled files (.so, .pdf, etc.) in the tarball without sources (.rest, .tex, ...). For example, if you are using Sphinx to generate documentation, ship conf.py and the .rst or .txt source files, not the generated HTML. A post installation script can do the HTML generation.

  • Do not copy other modules in the source tree of your application. Providing security support for such modules is a nightmare. If you’re missing something in foo module, send its upstream author a patch instead.

  • Never depend on an unreleased version of any library.

  • When you fix a bug, don’t re-release a modified tarball with the same version. Always increment the version.

  • Before releasing a version of your application, it is recommended to check your source code automatically for example with pyflakes:

    $ pyflakes phatch/ | grep -v "undefined name '_'" | grep -v 'but unused'| grep -v redefinition

Debian

Getting your package in the official Debian repository has a big advantage as many Linux distributions derive from it (Ubuntu, Xandros, Mepis, ...). So if you want to package your application for Ubuntu, it means you want to package it for Debian. Debian always has at least three distributions in active maintenance: stable, testing and unstable. Ubuntu is usually based on Debian unstable, future Long Term Support (LTS) versions are based on Debian testing. A Debian source package has the .dsc extension, while a binary package has the .deb extension. (Technically a .dsc file glues all files together: one or multiple tarballs with upstream sources and with Debian modifications, which have mostly the .diff.gz or .debian.tar.gz extension. All of them together are a “source package”.)

Where to Start

Linux uses a river metaform in which ‘upstream’ refers to the original software and its developers, and ‘downstream’ refers to packaging an application for distribution. (According to the metaform of a ‘stream’, Debian is also considered upstream for Ubuntu.) The Python Applications Packaging Team (PAPT) is the Debian packaging team of Python programs that don’t provide public modules. (If you want to distribute a python module, contact the Debian Python Module Packaging Team.) PAPT prefers upstream authors to work with them if you wish to have your code in Debian. This is more desirable than creating unofficial packages which are never uploaded to Debian.

If you don’t know how to make Debian packages, the first thing to do is report a Request For Packaging (RFP) bug by sending a mail to submit@bugs.debian.org which follows the following template:

  • Subject of email:

    RFP: phatch -- simple to use Photo Batch Processor
  • Body of email:

    Package: wnpp
    Severity: wishlist
    
    * Package name    : phatch
      Version         : 0.2.1
      Upstream Author : Stani M <spe.stani.be@gmail.com>
    * URL             : http://photobatch.stani.be
    * License         : GPL
      Programming Lang: Python
      Description     : simple to use Photo Batch Processor
    
    Phatch handles all popular image formats and can duplicate (sub)directory
    hierarchies. It can batch resize, rotate, apply perspective, shadows, rounded
    corners, ... and more in minutes instead of hours or days if you do it
    manually. Phatch allows you to use EXIF and IPTC tags for renaming and data
    stamping.

Submitting a RFP gives no guarantee that your application gets packaged. In order to speed up the process, do everything you can to prepare the package. Look for packages with similar requirements in the PAPT packages sources and use these as a base for your own package. Also read and understand the Debian Python policy, for example:

  • Your application must specify a dependency on Python, with a versioned dependency if necessary.
  • Private modules should be installed in /usr/share/package_name/module, or /usr/lib/package_name/module if the modules are architecture-dependent (e.g. extensions)
  • Byte-compiled modules must not be shipped with the binary package, they should be generated in the package’s post-install script, and removed in the pre-remove maintainer script.

After you’ve prepared the package, contact the PAPT team and there will be a high chance they will finish the package for you and upload it to the repositories. You might want to join PAPT to continue assisting with future releases of your application.

If you know how to make Debian packages, you can report an Intend To Package (ITP) bug. The template for the bug looks exactly the same as for RFP bugs except “RFP” is replaced with “ITP” in the Subject header. Others might offer help or ask about the status of work be replying to NNNNNN@bugs.debian.org (so just like in any other Debian bug). ITP is not a must, but it’s polite to tell others that you’re working on something so that they’ll not waste their time by duplicating the work. ITP bugs are also sent to debian-devel mailing list by default so other Debian Developers might let you know that you’re wasting your time because there are some copyright issues or any other problems with the software. When you are ready, ask PAPT on debian-python mailing list or #debian-python IRC channel (OFTC network) to check the source .dsc package and upload it to the official repositories. Mention that you are the upstream author.

Useful Tools

Debian packaging is described by files in a debian directory. For example the debian/control contains all the dependencies and a description. The following tools helps you to create a template for the debian directory:

  • dh_make: A command line utility that generates templates for all the files needed to build a Debian package. You still need to edit most of the template files to build the package.
  • debhelper: A collection of programs that can be used in a Debian/rules file to automate common tasks related to building binary Debian packages. Programs are included to install various files into your package, compress files, fix file permissions, integrate your package with the Debian menu system, etc.

The following are Python specific tools:

  • python-stdeb: Adds a new distutils command,``sdist_dsc``, to produce Debian source packages from Python packages. An additional command, bdist_deb, creates a Debian binary package, a .deb file. This package is going to be available in the future Debian and Ubuntu releases. Two convenience utilities are also provided:
    • pypi-install will query the Python Package Index (PyPI) for a package and download it. Afterwards it creates a .deb package, which it installs.
    • py2dsc will convert a distutils-built source tarball into a Debian source package.
  • python-mkdebian in the python-distutils-extra package similar to python-stdeb
  • python-central, python-support: Both of them integrate with debhelper and manage byte-compilation, private modules and dependencies among other things.

These are some examples from the python-stdeb documentation:

$ pypi-install mypackage # install from PyPi

$ python setup.py --command-packages=stdeb.command bdist_deb # make a deb binary package

$ py2dsc source.tar.gz # make a dsc source package

$ # build a deb binary from the dsc source
$ cd deb_dist/reindent-0.1.0/
$ dpkg-buildpackage -rfakeroot -uc -us

During the package process you will run into more abbreviations such as PAPT and RFP. You can look them up in this glossary or install the wtf tool:

$ sudo apt-get install bsdgames
$ wtf apt
apt: apt (8)              - Advanced Package Tool
Distributing on Ubuntu

It is recommended to submit your application to the Debian repositories through PAPT for Ubuntu distribution. Depending on the release schedule it will be synced from Debian to Ubuntu automatically or after a sync request. Ubuntu has a release cycle of 6 months. The last two months there is a Feature Freeze, during which no new packages or updates are allowed.

In case you want to test your application directly with an audience of Ubuntu users without any bureaucracy, Ubuntu’s Personal Package Archives (PPA) are the right solution. A PPA is a system to distribute software and updates directly to Ubuntu users. It uses Launchpad, which is a collaboration website like Sourceforge or Google Code. Create your source package, upload it and Launchpad will build binaries and then host them in your own apt repository:

  • This is the easiest way to distribute to end users during development.
  • Users will automatically receive updates as and when you make them.
  • PPA system only accepts source-only uploads.
  • Outsource the binary builds for the different Ubuntu versions.
  • PPA build system supports i386, amd64 and lpia architectures.
  • It is recommended to use one PPA for one application

We use PPA’s for Phatch for experimental development versions. Once ‘stable’ enough we push it into Debian Unstable. To upload to Debian Unstable you need permission of somebody else, while a PPA is under your own control (you can also remove packages). To upload your package use the command dput:

$ dput ppa:stani/ppa <source.changes>

Starting to Karmic/9.10, end user can add the PPA and its key as simple as:

$ sudo add-apt-repository ppa:stani

The Ubuntu community is trying to build an ecosystem for the opportunistic developers around Launchpad and Bazaar with the following tools:

  • Quickly provides a simple way of generating a new application, changing the GUI, saving it to bazaar, uploading the code to Launchpad and releasing your program to others in a Personal Package Archive.
  • Acire provides a library of Python snippets which you can study and run.
  • Ground Control: It is highly contextual graphical frontend for Bazaar and Launchpad, which tries to eliminate the need for using the command line. It is built into the Nautilus file manager and it only ever shows you a button for what you can do at that time.

For more information attend the Ubuntu Opportunistic Developer Week.

RPM-based Linux Distributions

RPM Format

The RPM format is used by many popular Linux distributions, including Red Hat, Fedora, SUSE, CentOS and Mandriva. distutils has a good basic support for the RPM format using the bdist_rpm command:

python setup.py bdist_rpm

However, since each distribution has its own path specifications and packaging policies, the resulting RPM will most likely not be accepted into the distribution repository. Therefore we suggest you carefully read about the packaging process first and contact the list of packagers if you intend to distribute your application to end users.

An RPM build for Fedora can be used on SUSE. (SUSE on Fedora is also possible). However, if your binary is linked to a library that is not available on the other distribution, that binary will not execute. Also, some path locations may be different.

Build Services

Red Hat, Fedora and CentOS use Koji as the platform to build RPM’s and do packaging tests. The Koji wiki explains the process. OpenSUSE has it is own build service. It claims to be the only service that allows developers to package software for all major Linux distributions.`

Create Your Own MIME Type

Include an .xml file that describes your MIME Type and copy it to /usr/share/mime/packages. For example phatch.xml:

<?xml version="1.0" encoding="UTF-8"?>
<mime-info xmlns="http://www.freedesktop.org/standards/shared-mime-info">
  <mime-type type="application/x-phatch">
    <sub-class-of type="text/plain"/>
    <comment>Phatch action list</comment>
    <comment xml:lang="nl">Phatch actielijst</comment>
    <glob pattern="*.phatch"/>
  </mime-type>
</mime-info>

Future

Guido accepted these two PEPs recently related to distutils:

Credits

  • Ayman Hourieh
  • Nele Decock
  • Nicoleau Fabien (Fedora)
  • Piotr Ożarowski (Debian)
  • Robin Mills

Table Of Contents

Previous topic

Release Manager

Next topic

actions