1
Scaled Window Support in DMX
7
15 October 2003 (created 19 September 2003)
9
This document investigates the possibility of adding scaled window support to
10
the DMX X server, thereby allowing a window or some selected part of the
11
logical DMX area to be displayed using a scaling factor. For example, this
12
might allow the contents of a window to be magnified for easier viewing. In
13
particular, scaling for the VNC client is explored. Copyright 2003 by Red Hat,
14
Inc., Raleigh, North Carolina
16
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
34
Application-transparent Scaling for DMX
35
XCreateScaledWindow API
37
Conclusion and Recommendations
43
The DMX X server (Xdmx) is a proxy server that is designed to allow X servers
44
on multiple machines to be combined into a single multi-headed X server.
45
Combined with Xinerama, these heads can appear as a single very high-resolution
46
screen. Typical applications include the creation of a video wall with 16
47
1280x1024 displays arranged in a rectangle, for a total resolution of of
52
Applications displayed on a physically large video wall that provides high
53
pixel-resolution may be difficult to see, especially if the application is
54
designed for use on a typical desktop computer with a relatively small display
55
located close to the human operator. The goal of this paper is to describe and
56
discuss solutions to this problem.
58
The original driving problem for this work is to provide scaling for the
59
vncviewer application when displayed using DMX (VNC scaling is currently
60
available only with the Windows client, and there is no plan to extend that
61
capability to other clients). While this specific problem will be addressed in
62
this paper, the general solution space will also be explored, since this may
63
lead to a good solution not only for vncviewer but also for other applications.
67
For reference, here is the original description of the task this paper
70
● Scaled window support (for VNC)
72
○ Investigate possibility of implementing a "scaled window" extension:
74
□ Add XCreateScaledWindow call that could be used in place of
77
□ All primitives drawn to scaled window would be scaled by
78
appropriate (integral?) scaling factor
80
○ Alternate approach: special case VNC support
84
This section reviews relevant previous work.
90
When using the vncviewer program for Windows, it is possible to specify a
91
scaling factor (as numerator and denominator). When scaling is in effect, the
92
viewer software uses StretchBlt (instead of BitBlt) to display the pixels for
93
the user. When this call is made, the viewer already has received all of the
94
pixel information (at full unscaled resolution).
96
The scaling in VNC is primitive. It does not conserve bandwidth, it does not
97
treat textual information differently (i.e., by using a suitably scaled font),
98
and it does not provide any anti-aliasing other than that provided by the
99
underlying (Windows-only) system library.
101
The X Video Extension
103
The X Video Extension is a widely-available extension to the X11 protocol that
104
provides support for streaming video. Integral to this support is the ability
105
to arbitrarily scale the output. In version 2.2 of the X Video specification,
106
support for scaled still images was provided, using both shared memory and
107
traditional transport. The API for this support uses calls that are quite
108
similar to XCreateWindow, XPutImage, and XShmPutImage. Currently, most of the
109
drivers implemented in XFree86 only support data in various YUV formats.
110
However, several modern video adaptors support RGB as well.
112
Note, though, that the target output for this scaling is an overlay plane -- so
113
X Video provides functionality that is fundamentally different from that
114
provided by the Windows StrechBlt call.
118
This section briefly discusses possible solutions, including major advantages
119
and disadvantages from both the implementation and the end-user programmer
126
The vncviewer application could be modified to provide software scaling. This
127
is not a general solution, but it does solve one of the goals of this work.
129
A prototype of this solution was implemented and a patch against
130
vnc-3.3.7-unixsrc is available in the dmx/external directory. Because of
131
limited time available for this work, all of the edge cases were not considered
132
and the solution works well mainly for integer scaling.
134
Currently, vncviewer writes to the X display with XPutImage, XCopyArea, and
135
XFillRectangle. All instances of these calls have to be aware of scaling and
136
must round correctly. In the prototype solution, rounding is incorrect and can
139
A better solution would be to cache all updates to the desktop image in
140
vncviewer and only send the damaged area to the X display with XPutImage. This
141
would allow the damaged area to be computed so that rounding errors do not
142
create artifacts. This method is probably similar to what is used in the Window
143
client. (The whole VNC suite is being re-written in C++ and the forthcoming
144
version 4 has not been evaluated.)
146
Scaling with the X Video Extension
148
The scaling in the Windows vncviewer application makes use of a scaled blit
149
that is supplied by the underlying system library. Several video cards
150
currently provide support for a scaled blit, and some X servers (including
151
XFree86) expose this capability to applications via the XvPutImage interface of
152
the X Video Extension. The capability exposed by XvPutImage results in the
153
scaled image being drawn to an overlay plane. Most video cards also provide
154
support for a scaled blit into the normal output planes, but this is not
155
exposed via XvPutImage.
157
The vncviewer program could be modified to use the X Video Extension to provide
158
scaling under X11 that is similar to the scaling currently provided under
159
Windows. Unfortunately, Xdmx does not currently export the X Video Extension,
160
so this would not provide an immediate solution usable with DMX.
162
A very early-stage proof-of-concept prototype was implemented and a preliminary
163
patch against vnc-3.3.7-unixsrc is available in the dmx/external directory.
164
This prototype was implemented to better understand the problems that must be
165
solved to make this solution viable:
167
● As noted under the software scaling section above, vncviewer writes to the
168
X display with several different calls. These calls write to the normal
169
output planes and are compatible with XvPutImage, which writes to an
170
overlay plane. To eliminate artifacts caused by this problem, vncviewer
171
should be modified so that a cached copy of the desktop is available,
172
either as a client-side image or a server-side off-screen pixmap, so that
173
XvPutImage would be the only method for writing to the X display.
175
● Although several modern graphics adaptors support hardware scaling using an
176
RGB format (e.g., ATI Radeon, nVidia, etc.), XFree86 drivers typically only
177
implement YUV formats. YUV generally compress the pixel information in some
178
way. For example, two commonly implemented formats, YUY2 and UYVY provide
179
intensity information for every RGB pixel, but only provide chroma and
180
luminance information for pairs of horizontal pixels. Since VNC uses
181
pixel-resolution for communicating updates on the wire, additional
182
artifacts are introduced (because there may not be enough information from
183
the wire to update a pair of pixels).
185
Further, the well-known problem with YUV encoding is even more evident when
186
the image is a desktop instead of a movie. For example, consider a
187
1-pixel-wide vertical window border. If the border changes in color but not
188
intensity (e.g., because a window manager uses color to indicate focus),
189
there may or may not be a change in the YUY2 image, depending on the
190
algorithm used for RGB to YUV conversion and on how the border pixel is
191
ordered in the pair of pixels used by the algorithm.
193
Many of these artifacts could be eliminated if vncviewer cached a complete
194
RGB image of the desktop, and only did the conversion to YUV for properly
195
aligned areas of damage. The remaining artifacts could be eliminated if an
196
RGB format was used with X Video (which may require the extension of
197
existing XFree86 drivers to support RGB).
199
● Most modern video cards support exactly one overlay plane that is suitable
200
for use with X Video. Therefore, only one application can use X Video at
201
any given time. This is a severe limitation in a desktop environment.
203
Implementing the X Video Extension for DMX
205
The user-level API for X Video is fairly simple, but the underlying support
206
required for the full specification is large. However, since the API provides a
207
method to query supported capabilities, a usable subset of X Video can be
208
implemented that would support XvPutImage and little else. This would require
209
support for the following:
211
● X Video Extension API calls, including the following:
217
○ XvQueryPortAttributes
233
● Support for querying back-end X Video Extension capabilities.
235
● Support for sending the image to the back-ends. Because X Video requires
236
sending full images, there may be a trade-off between bandwidth limitations
237
and additional complexity to divide the image up such that is scales
240
● Possible support for a software fall-back. For example, if all of the
241
back-ends do not support the X Video Extension, software scaling can be
242
implemented such that the image is sent to the back-end with XPutImage.
243
This pathway would have poor performance.
245
Supporting RGB formats for the X Video Extension
247
Assuming an XFree86 driver already supports the X Video Extension, and assuming
248
the target hardware supports an RGB format, then adding support for that format
249
is relatively simple and straightforward.
251
Scaling with an XPutImageScaled Extension
253
Instead of (or in addition to) implementing the X Video Extension in DMX, one
254
obvious solution would be to implement a new extension that provides access to
255
hardware-assisted scaled blits, similar to the StretchBlt call available under
256
Windows. This call would scale RGB images and would not use the overlay plane
257
(unlike the X Video Extension).
259
This approach has many of the same advantages and disadvantages as the
260
XCopyAreaScaled Extension, discussed in the next section. Discussion of
261
XPutImageScaled is deferred in favor of XCopyAreaScaled for the following
264
● XPutImageScaled can be emulated with XCopyAreaScaled by first using
265
XPutImage to copy the image to an off-screen pixmap, and then calling
266
XCopyAreaScaled between that off-screen pixmap and the target drawable.
268
● Since XCopyAreaScaled would copy between two areas of on-screen or
269
off-screen memory, it has additional uses and can be viewed as efficiently
270
providing a superset of XPutImageScaled functionality.
272
Scaling with an XCopyAreaScaled Extension
274
As noted in the previous section, because XCopyAreaScaled provides a superset
275
of the functionality provided by XPutImageScaled, we will consider this
278
First, XCopyAreaScaled would provide for RGB scaling between pixmaps (i.e.,
279
on-screen or off-screen areas of memory that reside on the video card). Unlike
280
the X Video Extension, which writes into an overlay plane, XCopyAreaScaled
281
would write into the non-overlay areas of the screen. Key points to consider
284
● Because different planes are involved, the two scaling operations are
285
usually implemented in hardware differently, so an XCopyAreaScaled
286
extension could be added in a manner that would neither conflict with nor
287
interact with the X Video extension in any way.
289
● The XCopyAreaScaled extension provides new functionality that the X Video
290
Extension does not provide. Based on anecdotal feedback, we believe that
291
many people outside the DMX and VNC communities would be excited about this
294
● The main drawback to this extension is that it is new and needs to be
295
implemented at the driver level in XFree86 for each video card to be
296
supported. At the present time, it is more likely that the X Video
297
Extension will be implemented for a particular piece hardware because the X
298
Video extension has multimedia uses. However, over time, we would expect
299
the XCopyAreaScaled extension to be implemented along with the X Video
300
extension, especially if it becomes popular.
302
● Another drawback is that not all modern cards provide support for a simple
303
scaled blit operation. However, these cards usually do provide a 3D
304
pipeline which could be used to provide this functionality in a manner that
305
is transparent to the client application that is using the XCopyAreaScaled
306
extension. However, this implementation pathway would make this extension
307
somewhat more difficult to implement on certain cards.
311
Another general solution to the scaling problem is to use the texture scaling
312
found in all 3D hardware. This ability is already exposed through OpenGL and
313
can be exploited by clients without X server modification (i.e., other than the
314
ability to support OpenGL). An application using OpenGL would transmit the
315
non-scaled image to the X server as a texture, and would then display a single
316
non-transformed rect using that texture. This also works around the single
317
overlay problem with the X Video Extension as well as the need to implement
318
additional scaled primitive extensions.
320
The downside is that most OpenGL implementations require power of 2 texture
321
sizes and this can be very wasteful of memory if, for example, the application
322
needs to scale a 1025x1025 image, which would require a 2048x2048 texture area
323
(even a 640x480 image would require a 1024x512 texture). Another downside is
324
that some OpenGL implementations have a limited about of texture memory and
325
cannot handle textures that are very large. For example, they might limit the
326
texture size to 1024x1024.
328
Application-transparent Scaling for DMX
330
Back-end Scaling Without Disconnect/Reconnect
332
VNC does scaling on the client side (in the vncviewer application).
333
Implementing a similar solution for DMX would require support in the back-end X
334
servers and, therefore, is not a general solution.
336
XFree86 already implements some support for "scaling" that could be used with
337
DMX: if, in the XF86Config file, multiple Modes are listed in the Display
338
Subsection of the Screen Section, then pressing Ctrl-Alt-Plus and
339
Ctrl-Alt-Minus can be used to iterate through the listed modes. The display
340
dimensions will change to the dimensions in the Modes line, but the logical
341
dimensions of the X server (i.e., the dimensions that Xdmx knows about) will
344
Further, the dimensions of the XFree86 display are under software control (via
345
the XFree86-VidModeExtension), so the Xdmx server could change the screen
346
dimensions on a per-display basis, thereby scaling the information on part of
349
However, this scaling appears to have limited use. For example, assume a 4 by 4
350
display wall consisting of 16 1280x1024 displays. If all of the back-end
351
servers were simultaneously configured to display 640x480, the left hand corner
352
of each display would be magnified, but the composite result would be
353
unreadable. Magnifying one display at a time could be usable, but could have
354
limited utility, since the result would still be no larger than a single
357
Back-end Scaling With Disconnect/Reconnect
359
Disconnect and reconnect features are not currently supported in DMX, but are
360
scheduled to be implemented in the future. These features, combined with the
361
XFree86-VidModeExtension Extension, would allow an application to do the
364
● Disconnect a specific back-end server (via the DMX Extension),
366
● reconfigure the XFree86 back-end server resolution, and
368
● reconnect the back-end server to DMX -- at a new origin with the new screen
371
For example, consider a display wall consisting of 16 1280x1024 displays with a
372
total resolution of 5120x4096. All of the screens could be disconnected,
373
repositioned, and reconnected each at a resolution of 640x480. The total
374
resolution of the display wall would be 2560x1920, allowing a view of a
375
selected area approximately one-fourth of the size of the DMX display. This
376
change would be completely application independent (except, perhaps, for a
377
DMX-aware window manager). When work at the increased resolution was completed,
378
the back-end servers could be disconnected, reconfigured, and reconnected for
379
the original 5120x4096 view.
381
Support for this type of scaling can be implemented in a DMX-aware X11 client
382
assuming the DMX server support arbitrary disconnect and reconnect semantics.
383
Because this application cannot be written before disconnect/reconnect is
384
implemented, this solution will not be discussed further in this paper.
388
In earlier versions of DMX, a frame buffer was maintained on the server side,
389
and XPutImage was used to move the information from the server to the client
390
(similar to some early VNC implementations). The use of a server-side frame
391
buffer would allow the server to do scaling, but is not a recommended solution
392
because of overall performance issues and server-side memory issues (i.e., the
393
frame buffer would be very large for large display walls).
395
Exploration of this path is not recommended.
397
XCreateScaledWindow API
399
The implementation of X Video Extension in DMX, and the use of XvPutImage by
400
applications requiring scaling requires significant changes in DMX Further,
401
XvPutImage is, essentially a scaled blit, and it is only useful for
402
applications which are already using (or can be modified to use) XPutImage.
403
Therefore, a more general API will be discussed as another possibility.
405
X applications typically create windows with the XCreateWindow call. A new
406
extension could provide an XCreateScaledWindow call that could be used in place
407
of the XCreateWindow call and be otherwise transparent to the application. This
408
would allow applications, even those that do not depend on XPutImage, to take
409
advantage of window scaling. In this section we describe how the call would
410
work, what transparency it provides, and how to solve the potential problems
411
that transparency creates.
415
The XCreateWindow call takes width and height as parameters. An
416
XCreateScaledWindow call could take all the same parameters, with the addition
421
An X11 window has several attributes that would have to be scaled:
423
● Background and border pixmaps
429
XGetWindowAttributes, XGetGeometry
431
For transparency, calls that query the window attributes should return unscaled
432
information. This suggests that all unscaled pixmaps and window attributes
435
Unfortunately, a window manager requires the scaled geometry to properly
436
decorate the window. The X server can probably determine which client is acting
437
as the window manager (e.g., because that client will select events that are
438
used exclusively by the window manager). However, other Scaled Window Extension
439
aware clients may also need to determine the scaled geometry. Therefore, at
440
least two additional extension calls should be implemented:
441
XGetScaledWindowAttributes and XGetScaledGeometry.
443
Popup and Child window positions
445
Some applications may position popup and child windows based on an unscaled
446
notion of the main window geometry. In this case, additional modifications to
447
the client would be required.
451
Most events (e.g., for mouse motion) return information about the coordinates
452
at which the even occurred. These coordinates would have to be modified so that
453
unscaled values were presented to the client.
457
There are many implementation issues, some of which are similar to the issues
458
involved in implementing the X Video Extension for DMX. The window contents
459
must be scaled, either by performing all operations to a frame buffer and then
460
writing the image to the display (perhaps using hardware scaling support), or
461
by modifying all of the various drawing operations to perform scaling. Because
462
of the complexity involved, the frame buffer option is recommended.
464
Conclusion and Recommendations
466
We recommend a three phase implementation strategy, based on how an application
467
could be written to take advantage of scaling:
469
1. The XCopyAreaScaled extension should be implemented, since this is the
470
ideal solution for applications like VNC, and since making use of this
471
extension will require minimal changes to applications that already use
472
XPutImage or XCopyArea.
474
The initial implementation work would include the design of the X protocol
475
extension, writing this up in the usual format for extension documentation,
476
implementation of the protocol transport pieces in XFree86, implementation
477
of a software fall-back in XFree86 and DMX, one example hardware
478
implementation for XFree86, and implementation of support for this
481
We suggest implementing the extension first on the ATI Radeon cards.
482
However, since these cards do not provide a 2D scaled blit primitive, the
483
implementation would have to make use of the 3D texture engine to emulate a
484
scaled blit. This is recommended, since other modern graphics cards also do
485
not provide a simple 2D scaled blit operation and an example of the more
486
difficult implementation pathway would be helpful to others.
488
2. Until XCopyAreaScaled is widely supported, applications that require
489
scaling will have to fall back to another scaling method. We suggest OpenGL
490
as the first fall-back method because it is widely available and supported
493
A project centered around OpenGL-based scaling would implement this scaling
494
in VNC as an example. This work would include re-writing the vncviewer
495
rendering engine to cache a master copy of the desktop image for all
498
3. Since OpenGL is not implemented everywhere, and may not provide
499
hardware-assisted performance in every implementation, an application that
500
requires scaling should also fall back to using the X Video Extension.
502
This project would add support for the X Video Extension to DMX and would
503
add support to VNC to take advantage of this extension without introducing
504
artifacts. This would require modifying the vncviewer rendering engine to
505
cache a master copy of the desktop image for all operations. This project
506
should also add support for the RGB format to at least one XFree86 driver
509
The X Video Extension is one of the few popular extensions that DMX does
510
not support. We recommend implementing the X Video Extension even if
511
scaling is the specific goal of that work.
513
We do not recommend implementation of the XCreateScaledWindow extension because
514
of the complexity involved. We do not recommend implementation of the
515
XPutImageScaled extension because it requires the same amount of work as the
516
XCopyAreaScaled extension, but provides less functionality. Further,
517
server-side scaling with a large frame buffer is not recommended because of the
518
performance implications.
520
The back-end scaling, especially with disconnect/reconnect support should be
521
explored in the future after disconnect/reconnect is implemented, but not at
added
removed
hw/dmx/doc/scaled.txt
