1
Bazaar Windows Shell Extension Options
2
========================================
9
This document details the imlpementation strategy chosen for the
10
Bazaar Windows Shell Extensions, otherwise known as TortoiseBzr, or TBZR.
11
As justification for the strategy, it also describes the general architecture
12
of Windows Shell Extensions, then looks at the C++ implemented TortoiseSvn
13
and the Python implemented TortoiseBzr, and discusses alternative
14
implementation strategies, and the reasons they were not chosen.
16
The following points summarize the strategy.
18
* Main shell extension code will be implemented in C++, and be as thin as
19
possible. It will not directly do any VCS work, but instead will perform
20
all operations via either external applications or an RPC server.
22
* Most VCS operations will be performed by external applications. For
23
example, committing changes or viewing history will spawn a child
24
process that provides its own UI.
26
* For operations where spawning a child process is not practical, an
27
external RPC server will be implemented in Python and will directly use
28
the VCS library. In the short term, there will be no attempt to create a
29
general purpose RPC mechanism, but instead will be focused on keeping the
30
C++ RPC client as thin, fast and dumb as possible.
32
Background Information
33
----------------------
35
The facts about shell extensions
36
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
38
Well - the facts as I understand them :)
40
Shell Extensions are COM objects. They are implemented as DLLs which are
41
loaded by the Windows shell. There is no facility for shell extensions to
42
exist in a separate process - DLLs are the only option, and they are loaded
43
into other processes which take advantage of the Windows shell (although
44
obviously this DLL is free to do whatever it likes)
46
For the sake of this discussion, there are 2 categories of shell extensions:
48
* Ones that create a new "namespace". The file-system itself is an example of
49
such a namespace, as is the "Recycle Bin". For a user-created example,
50
picture a new tree under "My Computer" which allows you to browse a remote
51
server - it creates a new, stand-alone tree that doesn't really interact
52
with the existing namespaces.
54
* Ones that enhance existing namespaces, including the filesystem. An example
55
would be an extension which uses Icon Overlays to modify how existing files
56
on disk are displayed or add items to their context menu, for example.
58
The latter category is the kind of shell extension relevant for TortoiseBzr,
59
and it has an important implication - it will be pulled into any process
60
which uses the shell to display a list of files. While this is somewhat
61
obvious for Windows Explorer (which many people consider the shell), every
62
other process that shows a FileOpen/FileSave dialog will have these shell
63
extensions loaded into its process space. This may surprise many people - the
64
simple fact of allowing the user to select a filename will result in an
65
unknown number of DLLs being loaded into your process. For a concrete
66
example, when notepad.exe first starts with an empty file it is using around
67
3.5MB of RAM. As soon as the FileOpen dialog is loaded, TortoiseSvn loads
68
well over 20 additional DLLs, including the MSVC8 runtime, into the Notepad
69
process causing its memory usage to more than double - all without doing
70
anything tortoise specific at all.
72
This has wide-ranging implications. It means that such shell extensions
73
should be developed using a tool which can never cause conflict with
74
arbitrary processes. For this very reason, MS recommend against using .NET
75
to write shell extensions[1], as there is a significant risk of being loaded
76
into a process that uses a different version of the .NET runtime, and this
77
will kill the process. Similarly, Python implemented shell extension may well
78
conflict badly with other Python implemented applications (and will certainly
79
kill them in some situations). A similar issue exists with GUI toolkits used
80
- using (say) PyGTK directly in the shell extension would need to be avoided
81
(which it currently is best I can tell). It should also be obvious the shell
82
extension will be in many processes simultaneously, meaning use of a simple
83
log-file etc is problematic.
85
In practice, there is only 1 truly safe option - a low-level language (such
86
as C/C++) which makes use of only the win32 API, and a static version of the
87
C runtime library if necessary. Obviously, this sucks from our POV :)
89
[1]: http://blogs.msdn.com/oldnewthing/archive/2006/12/18/1317290.aspx
91
Analysis of TortoiseSVN code
92
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
94
TortoiseSVN is implemented in C++. It relies on an external process to
95
perform most UI (such as diff, log, commit etc commands), but it appears to
96
directly embed the SVN C libraries for the purposes of obtaining status for
97
icon overlays, context menu, drag&drop, etc.
99
The use of an external process to perform commands is fairly simplistic in
100
terms of parent and modal windows - for example, when selecting "Commit", a
101
new process starts and *usually* ends up as the foreground window, but it may
102
occasionally be lost underneath the window which created it, and the user may
103
accidently start many processes when they only need 1. Best I can tell, this
104
isn't necessarily a limitation of the approach, just the implementation.
106
Advantages of using the external process is that it keeps all the UI code
107
outside Windows explorer - only the minimum needed to perform operations
108
directly needed by the shell are part of the "shell extension" and the rest
109
of TortoiseSvn is "just" a fairly large GUI application implementing many
110
commands. The command-line to the app has even been documented for people who
111
wish to automate tasks using that GUI. This GUI appears to also be
112
implemented in C++ using Windows resource files.
114
TortoiseSvn appears to cache using a separate process, aptly named
115
TSVNCache.exe. It uses a named pipe to accept connections from other
116
processes for various operations. At this stage, it's still unclear exactly
117
what is fetched from the cache and exactly what the shell extension fetches
118
directly via the subversion C libraries.
120
There doesn't seem to be a good story for logging or debugging - which is
121
what you expect from C++ based apps :( Most of the heavy lifting is done by
122
the external application, which might offer better facilities.
124
Analysis of existing TortoiseBzr code
125
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
127
The existing code is actually quite cool given its history (SoC student,
128
etc), so this should not be taken as criticism of the implementer nor of the
129
implementation - indeed, many criticisms are also true of the TortoiseSvn
130
implementation - see above. However, I have attempted to list the bad things
131
rather than the good things so a clear future strategy can be agreed, with
132
all limitations understood.
134
The existing TortoiseBzr code has been ported into Python from other tortoise
135
implementations (probably svn). This means it is very nice to implement and
136
develop, but suffers the problems described above - it is likely to conflict
137
with other Python based processes, and it means the entire CPython runtime
138
and libraries are pulled into many arbitrary processes.
140
The existing TortoiseBzr code pulls in the bzrlib library to determine the
141
path of the bzr library, and also to determine the status of files, but uses
142
an external process for most GUI commands - ie, very similar to TortoiseSvn
143
as described above - and as such, all comments above apply equally here - but
144
note that the bzr library *is* pulled into the shell, and therefore every
145
application using the shell. The GUI in the external application is written
146
in PyGTK, which may not offer the best Windows "look and feel", but that
147
discussion is beyond the scope of this document.
149
It has a better story for logging and debugging for the developer - but not
150
for diagnosing issues in the field - although again, much of the heavy
151
lifting remains done by the external application.
153
It uses a rudimentary in-memory cache for the status of files and
154
directories, the implementation of which isn't really suitable (ie, no
155
theoretical upper bound on cache size), and also means that there is no
156
sharing of cached information between processes, which is unfortunate (eg,
157
imagine a user using Windows explorer, then switching back to their editor)
158
and also error prone (it's possible the editor will check the file in,
159
meaning Windows explorer will be showing stale data). This may be possible to
160
address via file-system notifications, but a shared cache would be preferred
161
(although clearly more difficult to implement)
163
One tortoise port recently announced a technique for all tortoise ports to
164
share the same icon overlays to help work around a limitation in Windows on
165
the total number of overlays (its limited to 15, due to the number of bits
166
reserved in a 32bit int for overlays). TBZR needs to take advantage of that
167
(but to be fair, this overlay sharing technique was probably done after the
170
The current code appears to recursively walk a tree to check if *any* file in
171
the tree has changed, so it can reflect this in the parent directory status.
172
This is almost certainly an evil thing to do (Shell Extensions are optimized
173
so that a folder doesn't even need to look in its direct children for another
174
folder, let alone recurse for any reason at all. It may be a network mounted
175
drive that doesn't perform at all)
177
Although somewhat dependent on bzr itself, we need a strategy for binary
178
releases (ie, it assumes python.exe, etc) and integration into an existing
181
Trivially, the code is not PEP8 compliant and was written by someone fairly
182
inexperienced with the language.
184
Detailed Implementation Strategy
185
---------------------------------
187
We will create a hybrid Python and C++ implementation. In this model, we
188
would still use something like TSVNCache.exe (this external
189
process doesn't have the same restrictions as the shell extension itself) but
190
go one step further - use this remote process for *all* interactions with
191
bzr, including status and other "must be fast" operations. This would allow
192
the shell extension itself to be implemented in C++, but still take advantage
193
of Python for much of the logic.
195
A pragmatic implementation strategy will be used to work towards the above
196
infrastructure - we will keep the shell extension implemented in Python - but
197
without using bzrlib. This would allow us to focus on this
198
shared-cache/remote-process infrastructure without immediately
199
re-implementing a shell extension in C++. Longer term, once the
200
infrastructure is in place and as optimized as possible, we can move to C++
201
code in the shell calling our remote Python process. This port should try and
202
share as much code as possible from TortoiseSvn, including overlay handlers.
204
External Command Processor
205
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
207
The external command application (ie, the app invoked by the shell extension
208
to perform commands) can remain as-is, and remain a "shell" for other
209
external commands. The implementation of this application is not particularly
210
relevant to the shell extension, just the interface to the application (ie,
211
its command-line) is. In the short term this will remain PyGTK and will only
212
change if there is compelling reason - cross-platform GUI tools are a better
213
for bazaar than Windows specific ones, although native look-and-feel is
214
important. Either way, this can change independently from the shell
217
Performance considerations
218
~~~~~~~~~~~~~~~~~~~~~~~~~~~
220
As discussed above, the model used by Tortoise is that most "interesting"
221
things are done by external applications. Most Tortoise implementations
222
show read-only columns in the "detail" view, and shows a few read only
223
properties in the "Properties" dialog - but most of these properties are
224
"state" related (eg, revision number), or editing of others is done by
225
launching an external application. This means that the shell extension itself
226
really has 2 basic requirements WRT RPC: 1) get the local state of a file and
227
2) get some named state-related "properties" for a file. Everything else can
230
There are 2 aspects of the shell integration which are performance critical -
231
the "icon overlays" and "column providers"
233
The short-story with Icon Overlays is that we need to register 12 global
234
"overlay providers" - one for each state we show. Each provider is called for
235
every icon ever shown in Windows explorer or in any application's FileOpen
236
dialog. While most versions of Windows update icons in the background, we
237
still need to perform well. On the positive side, this just needs the simple
238
"local state" of a file - information that can probably be carried in a
239
single byte. On the negative side, it is the shell which makes a synchronous
240
call to us with a single filename as an arg, which makes it difficult to
241
"batch" multiple status requests into a single RPC call.
243
The story with columns is messier - these have changed significantly for
244
Vista and the new system may not work with the VCS model (see below).
245
However, if we implement this, it will be fairly critical to have
246
high-performance name/value pairs implemented, as described above.
248
Note that the nature of the shell implementation means we will have a large
249
number of "unrelated" handlers, each called somewhat independently by the
250
shell, often for information about the same file (eg, imagine each of our
251
overlay providers all called in turn with the same filename, followed by our
252
column providers called in turn with the same filename. However, that isn't
253
exactly what happens!). This means we will need a kind of cache, geared
254
towards reducing the number of status or property requests we make to the RPC
257
We will also allow all of the above to be disabled via user preferences.
258
Thus, Icon Overlays could be disabled if it did cause a problem for some
264
Due to the high number of calls for icon overlays, the RPC overhead must be
265
kept as low as possible. Due to the client side being implemented in C++,
266
reducing complexity is also a goal. Our requirements are quite simple and no
267
existing RPC options exist we can leverage. It does not seen prudent to build
268
an XMLRPC solution for tbzr - which is not to preclude the use of such a
269
server in the future, but tbzr need not become the "pilot" project for an
270
XMLRPC server given these constraints.
272
I propose that a custom RPC mechanism, built initially using windows-specific
273
named-pipes, be used. A binary format, designed with an eye towards
274
implementation speed and C++ simplicity, will be used. If we succeed here, we
275
can build on that infrastructure, and even replace it should other more
276
general frameworks materialize.
278
FWIW, with a Python process at each end, my P4 2.4G machine can achieve
279
around 25000 "calls" per-second across an open named pipe. C++ at one end
280
should increase this a little, but obviously any real work done by the Python
281
side of the process will be the bottle-neck. However, this throughput would
282
appear sufficient to implement a prototype.
287
Let's try and avoid an OS advocacy debate :) But it is probably true that
288
TBZR will, over its life, be used by more Vista computers than XP ones. In
289
short, Vista has changed a number of shell related interfaces, and while TSVN
290
is slowly catching up (http://tortoisesvn.net/vistaproblems) they are a pain.
292
XP has IColumnProvider (as implemented by Tortoise), but Vista changes this
293
model. The new model is based around "file types" (eg, .jpg files) and it
294
appears each file type can only have 1 provider! TSVN also seems to think the
295
Vista model isn't going to work (see previous link). It's not clear how much
296
effort we should expend on a column system that has already been abandoned by
297
MS. I would argue we spend effort on other parts of the system (ie, the
298
external GUI apps themselves, etc) and see if a path forward does emerge for
299
Vista. We can re-evaluate this based on user feedback and more information
300
about features of the Vista property system.
305
The following is a high-level set of milestones for the implementation:
307
* Design the RPC mechanism used for icon overlays (ie, binary format used for
310
* Create Python prototype of the C++ "shim": modify the existing TBZR Python
311
code so that all references to "bzrlib" are removed. Implement the client
312
side of the RPC mechanism and implement icon overlays using this RPC
315
* Create initial implementation of RPC server in Python. This will use
316
bzrlib, but will also maintain a local cache to achieve the required
317
performance. The initial implementation may even be single-threaded, just
318
to keep synchronization issues to a minimum.
320
* Analyze performance of prototype. Verify that technique is feasible and
321
will offer reasonable performance and user experience.
323
* Implement C++ shim: replace the Python prototype with a light-weight C++
324
version. We would work from the current TSVN sources, including its new
325
support for sharing icon overlays. Advice on if we should "fork" TSVN, or
326
try and manage our own svn based branch in bazaar are invited.
328
* Implement property pages and context menus in C++. Expand RPC server as
331
* Create binary for alpha releases, then go round-and-round until its baked.
333
Alternative Implementation Strategies
334
-------------------------------------
336
Only one credible alternative strategy was identified, as discussed below. No
337
languages other than Python and C++ were considered; Python as the bzr
338
library and existing extensions are written in Python and otherwise only C++
339
for reasons outlined in the background on shell extensions above.
341
Implement Completely in Python
342
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
344
This would keep the basic structure of the existing TBZR code, with the
345
shell extension continuing to pull in Python and all libraries used by Bzr
346
into various processes.
348
Although implementation simplicity is a key benefit to this option, it was
349
not chosen for various reasons; The use of Python means that there is a
350
larger chance of conflicting with existing applications, or even existing
351
Python implemented shell extensions. It will also increase the memory usage
352
of all applications which use the shell. While this may create problems for a
353
small number of users, it may create a wider perception of instability or
added
removed
doc/developers/tortoise-strategy.txt
