1
<!DOCTYPE linuxdoc PUBLIC "-//XFree86//DTD linuxdoc//EN">
4
<!-- Title information -->
5
<title>Scaled Window Support in DMX</title>
6
<author>Rickard E. Faith and Kevin E. Martin</author>
7
<date>15 October 2003 (created 19 September 2003)</date>
9
This document investigates the possibility of adding scaled window
10
support to the DMX X server, thereby allowing a window or some
11
selected part of the logical DMX area to be displayed using a
12
scaling factor. For example, this might allow the contents of a
13
window to be magnified for easier viewing. In particular, scaling
14
for the VNC client is explored. <it>Copyright 2003
15
by Red Hat, Inc., Raleigh, North Carolina</it>
18
<!-- Table of contents -->
21
<!-- Begin the document -->
25
The DMX X server (Xdmx) is a proxy server that is designed
26
to allow X servers on multiple machines to be combined into
27
a single multi-headed X server. Combined with Xinerama,
28
these heads can appear as a single very high-resolution
29
screen. Typical applications include the creation of a
30
video wall with 16 1280x1024 displays arranged in a
31
rectangle, for a total resolution of of 5120x4096.
34
<sect1>Problem Statement
36
Applications displayed on a physically large video wall that
37
provides high pixel-resolution may be difficult to see,
38
especially if the application is designed for use on a
39
typical desktop computer with a relatively small display
40
located close to the human operator. The goal of this paper
41
is to describe and discuss solutions to this problem.
44
The original driving problem for this work is to provide
45
scaling for the <tt>vncviewer</tt> application when
46
displayed using DMX (VNC scaling is currently available only
47
with the Windows client, and there is no plan to extend that
48
capability to other clients). While this specific problem
49
will be addressed in this paper, the general solution space
50
will also be explored, since this may lead to a good
51
solution not only for <tt>vncviewer</tt> but also for
57
For reference, here is the original description of the task
60
<item>Scaled window support (for VNC)
63
Investigate possibility of implementing a "scaled
67
Add XCreateScaledWindow call that could be used
68
in place of XCreateWindow
71
All primitives drawn to scaled window would be
72
scaled by appropriate (integral?) scaling factor
77
Alternate approach: special case VNC support
88
This section reviews relevant previous work.
91
<sect2>Scaling under VNC
93
When using the <tt>vncviewer</tt> program for Windows, it
94
is possible to specify a scaling factor (as numerator and
95
denominator). When scaling is in effect, the viewer
96
software uses StretchBlt (instead of BitBlt) to display
97
the pixels for the user. When this call is made, the
98
viewer already has received all of the pixel information
99
(at full unscaled resolution).
102
The scaling in VNC is primitive. It does not conserve
103
bandwidth, it does not treat textual information
104
differently (i.e., by using a suitably scaled font), and
105
it does not provide any anti-aliasing other than that
106
provided by the underlying (Windows-only) system library.
110
<sect1>The X Video Extension
112
The X Video Extension is a widely-available extension to the
113
X11 protocol that provides support for streaming video.
114
Integral to this support is the ability to arbitrarily scale
115
the output. In version 2.2 of the X Video specification,
116
support for scaled still images was provided, using both
117
shared memory and traditional transport. The API for this
118
support uses calls that are quite similar to XCreateWindow,
119
XPutImage, and XShmPutImage. Currently, most of the drivers
120
implemented in XFree86 only support data in various YUV
121
formats. However, several modern video adaptors support RGB
125
Note, though, that the target output for this scaling is an
126
overlay plane -- so X Video provides functionality that is
127
fundamentally different from that provided by the Windows
133
<sect>Possible Solutions
135
This section briefly discusses possible solutions, including
136
major advantages and disadvantages from both the
137
implementation and the end-user programmer standpoint.
139
<sect1>VNC-like Scaling
140
<sect2>Software Scaling
142
The <tt>vncviewer</tt> application could be modified to
143
provide software scaling. This is not a general solution,
144
but it does solve one of the goals of this work.
147
A prototype of this solution was implemented and a patch
148
against <tt>vnc-3.3.7-unixsrc</tt> is available in the
149
<tt>dmx/external</tt> directory. Because of limited time
150
available for this work, all of the edge cases were not
151
considered and the solution works well mainly for integer
155
Currently, <tt>vncviewer</tt> writes to the X display
156
with XPutImage, XCopyArea, and XFillRectangle. All
157
instances of these calls have to be aware of scaling
158
and must round correctly. In the prototype solution,
159
rounding is incorrect and can cause artifacts.
162
A better solution would be to cache all updates to the
163
desktop image in <tt>vncviewer</tt> and only send the
164
damaged area to the X display with XPutImage. This would
165
allow the damaged area to be computed so that rounding
166
errors do not create artifacts. This method is probably
167
similar to what is used in the Window client. (The whole
168
VNC suite is being re-written in C++ and the forthcoming
169
version 4 has not been evaluated.)
172
<sect2>Scaling with the X Video Extension
174
The scaling in the Windows <tt>vncviewer</tt> application
175
makes use of a scaled blit that is supplied by the
176
underlying system library. Several video cards currently
177
provide support for a scaled blit, and some X servers
178
(including XFree86) expose this capability to applications
179
via the XvPutImage interface of the X Video Extension.
180
The capability exposed by XvPutImage results in the scaled
181
image being drawn to an overlay plane. Most video cards
182
also provide support for a scaled blit into the normal
183
output planes, but this is not exposed via XvPutImage.
186
The <tt>vncviewer</tt> program could be modified to use
187
the X Video Extension to provide scaling under X11 that is
188
similar to the scaling currently provided under Windows.
189
Unfortunately, Xdmx does not currently export the X Video
190
Extension, so this would not provide an immediate solution
194
A very early-stage proof-of-concept prototype was
195
implemented and a preliminary patch against
196
<tt>vnc-3.3.7-unixsrc</tt> is available in the
197
<tt>dmx/external</tt> directory. This prototype was
198
implemented to better understand the problems that must be
199
solved to make this solution viable:
202
As noted under the software scaling section above,
203
<tt>vncviewer</tt> writes to the X display with
204
several different calls. These calls write to the
205
normal output planes and are compatible with
206
XvPutImage, which writes to an overlay plane. To
207
eliminate artifacts caused by this problem,
208
<tt>vncviewer</tt> should be modified so that a cached
209
copy of the desktop is available, either as a
210
client-side image or a server-side off-screen pixmap,
211
so that XvPutImage would be the only method for
212
writing to the X display.
216
Although several modern graphics adaptors support
217
hardware scaling using an RGB format (e.g., ATI
218
Radeon, nVidia, etc.), XFree86 drivers typically
219
only implement YUV formats. YUV generally compress
220
the pixel information in some way. For example, two
221
commonly implemented formats, YUY2 and UYVY provide
222
intensity information for every RGB pixel, but only
223
provide chroma and luminance information for pairs
224
of horizontal pixels. Since VNC uses
225
pixel-resolution for communicating updates on the
226
wire, additional artifacts are introduced (because
227
there may not be enough information from the wire to
228
update a pair of pixels).
230
Further, the well-known problem with YUV encoding
231
is even more evident when the image is a desktop
232
instead of a movie. For example, consider a
233
1-pixel-wide vertical window border. If the border
234
changes in color but not intensity (e.g., because a
235
window manager uses color to indicate focus), there
236
may or may not be a change in the YUY2 image,
237
depending on the algorithm used for RGB to YUV
238
conversion and on how the border pixel is ordered in
239
the pair of pixels used by the algorithm.
241
Many of these artifacts could be eliminated if
242
<tt>vncviewer</tt> cached a complete RGB image of
243
the desktop, and only did the conversion to YUV for
244
properly aligned areas of damage. The remaining artifacts
245
could be eliminated if an RGB format was used with X
246
Video (which may require the extension of existing
247
XFree86 drivers to support RGB).
250
Most modern video cards support exactly one overlay
251
plane that is suitable for use with X Video.
252
Therefore, only one application can use X Video at any
253
given time. This is a severe limitation in a desktop
258
<sect3>Implementing the X Video Extension for DMX
260
The user-level API for X Video is fairly simple, but the
261
underlying support required for the full specification
262
is large. However, since the API provides a method to
263
query supported capabilities, a usable subset of X
264
Video can be implemented that would support XvPutImage
265
and little else. This would require support for the
269
X Video Extension API calls, including the
272
<item>XvQueryExtension</item>
273
<item>XvQueryAdaptors</item>
274
<item>XvQueryPortAttributes</item>
275
<item>XvFreeAdaptorInfo</item>
276
<item>XvListImageFormats</item>
277
<item>XvGrabPort</item>
278
<item>XvCreateImage</item>
279
<item>XvPutImage</item>
280
<item>XvShmCreateImage</item>
281
<item>XvShmPutImage</item>
285
Support for querying back-end X Video Extension
289
Support for sending the image to the back-ends.
290
Because X Video requires sending full images, there
291
may be a trade-off between bandwidth limitations and
292
additional complexity to divide the image up such
293
that is scales properly.
296
Possible support for a software fall-back. For
297
example, if all of the back-ends do not support the X
298
Video Extension, software scaling can be implemented
299
such that the image is sent to the back-end with
300
XPutImage. This pathway would have poor
306
<sect3>Supporting RGB formats for the X Video Extension
308
Assuming an XFree86 driver already supports the X Video
309
Extension, and assuming the target hardware supports an
310
RGB format, then adding support for that format is
311
relatively simple and straightforward.
315
<sect2>Scaling with an XPutImageScaled Extension
317
Instead of (or in addition to) implementing the X Video
318
Extension in DMX, one obvious solution would be to
319
implement a new extension that provides access to
320
hardware-assisted scaled blits, similar to the StretchBlt
321
call available under Windows. This call would scale RGB
322
images and would not use the overlay plane (unlike the X
326
This approach has many of the same advantages and
327
disadvantages as the XCopyAreaScaled Extension, discussed
328
in the next section. Discussion of XPutImageScaled is
329
deferred in favor of XCopyAreaScaled for the following
333
XPutImageScaled can be emulated with XCopyAreaScaled
334
by first using XPutImage to copy the image to an
335
off-screen pixmap, and then calling XCopyAreaScaled
336
between that off-screen pixmap and the target
340
Since XCopyAreaScaled would copy between two areas of
341
on-screen or off-screen memory, it has additional uses
342
and can be viewed as efficiently providing a superset
343
of XPutImageScaled functionality.
348
<sect2>Scaling with an XCopyAreaScaled Extension
350
As noted in the previous section, because XCopyAreaScaled
351
provides a superset of the functionality provided by
352
XPutImageScaled, we will consider this extension instead.
355
First, XCopyAreaScaled would provide for RGB scaling
356
between pixmaps (i.e., on-screen or off-screen areas of
357
memory that reside on the video card). Unlike the X Video
358
Extension, which writes into an overlay plane,
359
XCopyAreaScaled would write into the non-overlay areas of
360
the screen. Key points to consider are as follows:
363
Because different planes are involved, the two scaling
364
operations are usually implemented in hardware
365
differently, so an XCopyAreaScaled extension could be
366
added in a manner that would neither conflict with nor
367
interact with the X Video extension in any way.
370
The XCopyAreaScaled extension provides new
371
functionality that the X Video Extension does not
372
provide. Based on anecdotal feedback, we believe that
373
many people outside the DMX and VNC communities would
374
be excited about this extension.
377
The main drawback to this extension is that it is new
378
and needs to be implemented at the driver level in
379
XFree86 for each video card to be supported. At the
380
present time, it is more likely that the X Video
381
Extension will be implemented for a particular piece
382
hardware because the X Video extension has multimedia
383
uses. However, over time, we would expect the
384
XCopyAreaScaled extension to be implemented along with
385
the X Video extension, especially if it becomes
389
Another drawback is that not all modern cards provide
390
support for a simple scaled blit operation. However,
391
these cards usually do provide a 3D pipeline which
392
could be used to provide this functionality in a
393
manner that is transparent to the client application
394
that is using the XCopyAreaScaled extension. However,
395
this implementation pathway would make this extension
396
somewhat more difficult to implement on certain cards.
401
<sect2>Scaling with OpenGL
403
Another general solution to the scaling problem is to use
404
the texture scaling found in all 3D hardware. This
405
ability is already exposed through OpenGL and can be
406
exploited by clients without X server modification (i.e.,
407
other than the ability to support OpenGL). An application
408
using OpenGL would transmit the non-scaled image to the X
409
server as a texture, and would then display a single
410
non-transformed rect using that texture. This also works
411
around the single overlay problem with the X Video
412
Extension as well as the need to implement additional
413
scaled primitive extensions.
416
The downside is that most OpenGL implementations require
417
power of 2 texture sizes and this can be very wasteful of
418
memory if, for example, the application needs to scale a
419
1025x1025 image, which would require a 2048x2048 texture
420
area (even a 640x480 image would require a 1024x512
421
texture). Another downside is that some OpenGL
422
implementations have a limited about of texture memory and
423
cannot handle textures that are very large. For example,
424
they might limit the texture size to 1024x1024.
428
<sect1>Application-transparent Scaling for DMX
429
<sect2>Back-end Scaling Without Disconnect/Reconnect
431
VNC does scaling on the client side (in the
432
<tt>vncviewer</tt> application). Implementing a similar
433
solution for DMX would require support in the back-end X
434
servers and, therefore, is not a general solution.
437
XFree86 already implements some support for "scaling" that
438
could be used with DMX: if, in the XF86Config file,
439
multiple Modes are listed in the Display Subsection of the
440
Screen Section, then pressing Ctrl-Alt-Plus and
441
Ctrl-Alt-Minus can be used to iterate through the listed
442
modes. The display dimensions will change to the
443
dimensions in the Modes line, but the logical dimensions
444
of the X server (i.e., the dimensions that Xdmx knows
445
about) will not change.
448
Further, the dimensions of the XFree86 display are under
449
software control (via the XFree86-VidModeExtension), so
450
the Xdmx server could change the screen dimensions on a
451
per-display basis, thereby scaling the information on part
455
However, this scaling appears to have limited use. For
456
example, assume a 4 by 4 display wall consisting of 16
457
1280x1024 displays. If all of the back-end servers were
458
simultaneously configured to display 640x480, the left
459
hand corner of each display would be magnified, but the
460
composite result would be unreadable. Magnifying one
461
display at a time could be usable, but could have limited
462
utility, since the result would still be no larger than a
466
<sect2>Back-end Scaling With Disconnect/Reconnect
468
Disconnect and reconnect features are not currently
469
supported in DMX, but are scheduled to be implemented in
470
the future. These features, combined with the
471
XFree86-VidModeExtension Extension, would allow an
472
application to do the following:
475
Disconnect a specific back-end server (via the DMX
479
reconfigure the XFree86 back-end server resolution,
483
reconnect the back-end server to DMX -- at a new
484
origin with the new screen resolution.
489
For example, consider a display wall consisting of 16
490
1280x1024 displays with a total resolution of 5120x4096.
491
All of the screens could be disconnected, repositioned,
492
and reconnected each at a resolution of 640x480. The
493
total resolution of the display wall would be 2560x1920,
494
allowing a view of a selected area approximately
495
one-fourth of the size of the DMX display. This change
496
would be completely application independent (except,
497
perhaps, for a DMX-aware window manager). When work at
498
the increased resolution was completed, the back-end
499
servers could be disconnected, reconfigured, and
500
reconnected for the original 5120x4096 view.
503
Support for this type of scaling can be implemented in a
504
DMX-aware X11 client assuming the DMX server support
505
arbitrary disconnect and reconnect semantics. Because
506
this application cannot be written before
507
disconnect/reconnect is implemented, this solution will
508
not be discussed further in this paper.
511
<sect2>Server-side Scaling
513
In earlier versions of DMX, a frame buffer was maintained
514
on the server side, and XPutImage was used to move the
515
information from the server to the client (similar to some
516
early VNC implementations). The use of a server-side
517
frame buffer would allow the server to do scaling, but is
518
not a recommended solution because of overall performance
519
issues and server-side memory issues (i.e., the frame
520
buffer would be very large for large display walls).
523
Exploration of this path is not recommended.
527
<sect1>XCreateScaledWindow API
529
The implementation of X Video Extension in DMX, and the use
530
of XvPutImage by applications requiring scaling requires
531
significant changes in DMX Further, XvPutImage is,
532
essentially a scaled blit, and it is only useful for
533
applications which are already using (or can be modified to
534
use) XPutImage. Therefore, a more general API will be
535
discussed as another possibility.
538
X applications typically create windows with the
539
XCreateWindow call. A new extension could provide an
540
XCreateScaledWindow call that could be used in place of the
541
XCreateWindow call and be otherwise transparent to the
542
application. This would allow applications, even those that
543
do not depend on XPutImage, to take advantage of window
544
scaling. In this section we describe how the call would
545
work, what transparency it provides, and how to solve the
546
potential problems that transparency creates.
550
The XCreateWindow call takes width and height as
551
parameters. An XCreateScaledWindow call could take all
552
the same parameters, with the addition of a scaling factor.
555
<sect2>XSetWindowAttributes
557
An X11 window has several attributes that would have to be
560
<item>Background and border pixmaps</item>
561
<item>Border width</item>
566
<sect2>XGetWindowAttributes, XGetGeometry
568
For transparency, calls that query the window attributes
569
should return unscaled information. This suggests that
570
all unscaled pixmaps and window attributes should be
574
Unfortunately, a window manager requires the scaled
575
geometry to properly decorate the window. The X server
576
can probably determine which client is acting as the
577
window manager (e.g., because that client will select
578
events that are used exclusively by the window manager).
579
However, other Scaled Window Extension aware clients may
580
also need to determine the scaled geometry. Therefore, at
581
least two additional extension calls should be
582
implemented: XGetScaledWindowAttributes and
586
<sect2>Popup and Child window positions
588
Some applications may position popup and child windows
589
based on an unscaled notion of the main window geometry.
590
In this case, additional modifications to the client would
596
Most events (e.g., for mouse motion) return information
597
about the coordinates at which the even occurred. These
598
coordinates would have to be modified so that unscaled
599
values were presented to the client.
602
<sect2>Implementation
604
There are many implementation issues, some of which are
605
similar to the issues involved in implementing the X Video
606
Extension for DMX. The window contents must be scaled,
607
either by performing all operations to a frame buffer and
608
then writing the image to the display (perhaps using
609
hardware scaling support), or by modifying all of the
610
various drawing operations to perform scaling. Because of
611
the complexity involved, the frame buffer option is
618
<sect>Conclusion and Recommendations
620
We recommend a three phase implementation strategy, based on
621
how an application could be written to take advantage of
626
The XCopyAreaScaled extension should be implemented, since
627
this is the ideal solution for applications like VNC, and
628
since making use of this extension will require minimal
629
changes to applications that already use XPutImage or
632
The initial implementation work would include the design
633
of the X protocol extension, writing this up in the
634
usual format for extension documentation, implementation
635
of the protocol transport pieces in XFree86,
636
implementation of a software fall-back in XFree86 and
637
DMX, one example hardware implementation for XFree86,
638
and implementation of support for this extension in DMX.
640
We suggest implementing the extension first on the ATI
641
Radeon cards. However, since these cards do not provide
642
a 2D scaled blit primitive, the implementation would
643
have to make use of the 3D texture engine to emulate a
644
scaled blit. This is recommended, since other modern
645
graphics cards also do not provide a simple 2D scaled
646
blit operation and an example of the more difficult
647
implementation pathway would be helpful to others.
651
Until XCopyAreaScaled is widely supported, applications
652
that require scaling will have to fall back to another
653
scaling method. We suggest OpenGL as the first fall-back
654
method because it is widely available and supported by
657
A project centered around OpenGL-based scaling would
658
implement this scaling in VNC as an example. This work
659
would include re-writing the <tt>vncviewer</tt>
660
rendering engine to cache a master copy of the desktop
661
image for all operations.
665
Since OpenGL is not implemented everywhere, and may not
666
provide hardware-assisted performance in every
667
implementation, an application that requires scaling
668
should also fall back to using the X Video Extension.
670
This project would add support for the X Video Extension
671
to DMX and would add support to VNC to take advantage of
672
this extension without introducing artifacts. This
673
would require modifying the <tt>vncviewer</tt> rendering
674
engine to cache a master copy of the desktop image for
675
all operations. This project should also add support
676
for the RGB format to at least one XFree86 driver (e.g.,
679
The X Video Extension is one of the few popular
680
extensions that DMX does not support. We recommend
681
implementing the X Video Extension even if scaling is
682
the specific goal of that work.
687
We do <bf>not</bf> recommend implementation of the
688
XCreateScaledWindow extension because of the complexity
689
involved. We do <bf>not</bf> recommend implementation of the
690
XPutImageScaled extension because it requires the same amount
691
of work as the XCopyAreaScaled extension, but provides less
692
functionality. Further, server-side scaling with a large
693
frame buffer is <bf>not</bf> recommended because of the
694
performance implications.
697
The back-end scaling, especially with disconnect/reconnect
698
support should be explored in the future after
699
disconnect/reconnect is implemented, but not at the present
705
<!-- Local Variables: -->
706
<!-- fill-column: 72 -->
added
removed
hw/dmx/doc/scaled.sgml
