1
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
2
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
3
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
5
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
6
<meta name="generator" content="AsciiDoc 8.2.2" />
7
<style type="text/css">
9
p, li, dt, dd, div, pre, h1, h2, h3, h4, h5, h6 {
11
border: 1px solid red;
16
margin: 1em 5% 1em 5%;
21
text-decoration: underline;
39
h1, h2, h3, h4, h5, h6 {
41
font-family: sans-serif;
48
border-bottom: 2px solid silver;
51
border-bottom: 2px solid silver;
61
border: 1px solid silver;
76
font-family: sans-serif;
83
font-family: sans-serif;
87
font-family: sans-serif;
89
border-top: 2px solid silver;
95
padding-bottom: 0.5em;
99
padding-bottom: 0.5em;
103
div.tableblock, div.imageblock, div.exampleblock, div.verseblock,
104
div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
105
div.admonitionblock {
108
margin-bottom: 1.5em;
110
div.admonitionblock {
112
margin-bottom: 2.5em;
115
div.content { /* Block element content. */
119
/* Block element titles. */
120
div.title, caption.title {
121
font-family: sans-serif;
125
margin-bottom: 0.5em;
131
td div.title:first-child {
134
div.content div.title:first-child {
137
div.content + div.title {
141
div.sidebarblock > div.content {
143
border: 1px solid silver;
150
div.listingblock > div.content {
151
border: 1px solid silver;
156
div.quoteblock > div.content {
163
div.verseblock + div.attribution {
167
div.admonitionblock .icon {
171
text-decoration: underline;
173
padding-right: 0.5em;
175
div.admonitionblock td.content {
177
border-left: 2px solid silver;
180
div.exampleblock > div.content {
181
border-left: 2px solid silver;
185
div.verseblock div.content {
189
div.imageblock div.content { padding-left: 0; }
190
div.imageblock img { border: 1px solid silver; }
191
span.image img { border-style: none; }
195
margin-bottom: 0.8em;
207
list-style-position: outside;
210
list-style-type: lower-alpha;
213
div.tableblock > table {
214
border: 3px solid #527bbd;
217
font-family: sans-serif;
226
margin-bottom: 0.8em;
234
padding-right: 0.8em;
241
div#footer-badges { display: none; }
246
font-family: sans-serif;
250
margin-bottom: 0.1em;
253
div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
269
/* Workarounds for IE6's broken and incomplete CSS2. */
271
div.sidebar-content {
273
border: 1px solid silver;
276
div.sidebar-title, div.image-title {
277
font-family: sans-serif;
280
margin-bottom: 0.5em;
283
div.listingblock div.content {
284
border: 1px solid silver;
289
div.quoteblock-content {
293
div.exampleblock-content {
294
border-left: 2px solid silver;
298
/* IE6 sets dynamically generated links as visited. */
299
div#toc a:visited { color: blue; }
301
<title>Multibyte character processing in SigScheme</title>
305
<h1>Multibyte character processing in SigScheme</h1>
308
<div class="sectionbody">
309
<p>SigScheme's multibyte character handling interface is basically based on R6RS
310
Unicode character handlings. See also "R6RS conformance" section and
311
"Characters" subsection of "R5RS conformance" of <a href="spec.html">Specifications
312
of SigScheme</a>.</p>
313
<p>In addition to R6RS Unicode character handlings, SigScheme supports EUC-JP,
314
EUC-CN, EUC-KR and Shift_JIS character encoding schemes and they can be used
315
simultaneously. But no character encoding conversion method such as iconv is
318
<h2>2. Current character codec</h2>
319
<div class="sectionbody">
320
<p>On SigScheme, characters and strings are processed in accordance with the
321
parameter <strong>current character codec</strong>. Its initial value is specified by
322
<tt>—enable-default-encoding</tt> option of the configure script and defaults to
323
UTF-8. So characters and strings in SigScheme are treated as UTF-8 by default.</p>
324
<p>The value of current character codec can be checked by <tt>%%current-char-codec</tt>
325
procedure of SigScheme extension.</p>
326
<div class="listingblock">
327
<div class="content">
328
<pre><tt>sscm> (require-extension (sscm-ext))
329
sscm> (%%current-char-codec)
332
<p>To specify another encoding as default character codec, pass <tt>-C</tt> option to
333
<tt>sscm</tt> command as follows, or specify it in the second argument of
334
<tt>scm_initialize()</tt>.</p>
335
<div class="listingblock">
336
<div class="content">
337
<pre><tt>$ sscm -C ISO-8859-1
342
$ sscm -C Shift_JIS</tt></pre>
344
<p><tt>provided?</tt> predicate can be used to know whether an encoding is enabled by the
345
configuration or not.</p>
346
<div class="listingblock">
347
<div class="content">
348
<pre><tt>(provided? "utf8")
352
(provided? "sjis")</tt></pre>
354
<p>The <tt>ISO-8859-1</tt> encoding is used as generic singlebyte character encoding and
355
accepts any character represented in integer range 0-255, and always provided
356
regardless of configuration.</p>
357
<p>Use <tt>with-char-codec</tt> procedure to switch to another encoding temporarily.</p>
358
<div class="listingblock">
359
<div class="content">
360
<pre><tt>(define euc-A (with-char-codec "EUC-JP" (lambda () (integer->char #xa4a2))))</tt></pre>
362
<p>See also "Ports" section of this document for current character codec switching
365
<h2>3. Characters</h2>
366
<div class="sectionbody">
367
<p>When the reader is working on an UTF-8 port:</p>
368
<div class="listingblock">
369
<div class="content">
370
<pre><tt>#\あ ==> #\あ ;; U+3042 HIRAGANA LETTER A
371
#\x3042 ==> #\あ ;; U+3042 HIRAGANA LETTER A</tt></pre>
373
<p>Conversion between character and integer is performed based on the value of
374
<tt>%%current-char-codec</tt>. When <tt>%%current-char-codec</tt> is "UTF-8", integer value
375
of an Unicode character corresponds to the Unicode code point.</p>
376
<p>When <tt>%%current-char-codec</tt> is "UTF-8":</p>
377
<div class="listingblock">
378
<div class="content">
379
<pre><tt>(integer->char #x3042) ==> #\あ ;; U+3042 HIRAGANA LETTER A
380
(char->integer #\あ) ==> 12354 ;; #x3042</tt></pre>
382
<p>If an integer value is invalid for the current codec, an error is caused.</p>
383
<div class="listingblock">
384
<div class="content">
385
<pre><tt>(with-char-codec "UTF-8" (lambda () (integer->char #x3042))) ==> #\あ
386
(with-char-codec "ISO-8859-1" (lambda () (integer->char #x3042))) ==> error</tt></pre>
388
<p>Since a character in SigScheme is internally represented as an integer value
389
marked with character-object tag without any character encoding information,
390
user is responsible to manage the character encoding scheme of each character
392
<p>And no character encoding conversion is performed on integer<->char conversion
393
regaradless of <tt>%%current-char-codec</tt>.</p>
394
<div class="listingblock">
395
<div class="content">
396
<pre><tt>;; U+3042 HIRAGANA LETTER A in Unicode
397
(define ucs-A (with-char-codec "UTF-8" (lambda () (integer->char #x3042))))
399
;; HIRAGANA LETTER A in EUC-JP
400
(define eucjp-A (with-char-codec "EUC-JP" (lambda () (integer->char #xa4a2))))
402
(eqv? ucs-A eucjp-A) ==> #f
404
;; no conversion is performed
405
(with-char-codec "UTF-8"
406
(lambda () (char->integer eucjp-A))) ==> 42146 ;; U+A4A2 YI RADICAL ZUP</tt></pre>
410
<div class="sectionbody">
411
<p>When both reader's port and <tt>%%current-char-codec</tt> is UTF-8:</p>
412
<div class="listingblock">
413
<div class="content">
414
<pre><tt>"\x3042;a\x3044;" ==> "あaい"
415
(string->list "あaい") ==> (#\あ #\a #\い)
416
(string-length "あaい") ==> 3</tt></pre>
418
<p>A string in SigScheme is internally represented as a C string with its logical
419
character length without character encoding information. User is responsible to
420
manage the character encoding scheme of each string object. Though strings have
421
no encoding information, they have logical character length counted in
422
<tt>%%current-char-codec</tt> on its object creation. So processing a string in
423
another encoding such as UTF-8 string as byte string cannot fully be performed.</p>
424
<div class="listingblock">
425
<div class="content">
426
<pre><tt>;; U+3042 HIRAGANA LETTER A, with string length 1 counted in UTF-8
429
;; string length is not re-counted even if %%current-char-codec is changed
430
(with-char-codec "UTF-8" (lambda () (string-length utf8-A))) ==> 1
431
(with-char-codec "ISO-8859-1" (lambda () (string-length utf8-A))) ==> 1
433
;; character reference of string is based on %%current-char-codec
434
(with-char-codec "UTF-8" (lambda () (string-ref utf8-A 0))) ==> #\あ
435
(with-char-codec "ISO-8859-1" (lambda () (string-ref utf8-A 0))) ==> #\ã
437
;; character reference that exceeds logical length is an error even if its
438
;; physical length is enough
439
(with-char-codec "UTF-8" (lambda () (string-ref utf8-A 1))) ==> error
440
(with-char-codec "ISO-8859-1" (lambda () (string-ref utf8-A 1))) ==> error</tt></pre>
443
<h2>5. Identifiers</h2>
444
<div class="sectionbody">
445
<p>Any Unicode characters can be used as identifiers in Scheme as R6RS allows.</p>
446
<div class="listingblock">
447
<div class="content">
448
<pre><tt>'Français-symbole
450
(define ひらがな->カタカナ (lambda (イ . ロ) ...))</tt></pre>
452
<p>But since SigScheme's Unicode handling is incomplete, all non-ASCII Unicode
453
characters are treated as ordinary letter. Though it allows using any
454
whitespace characters and punctuations as identifier, it should not be done.</p>
455
<p>Non-ASCII identifier in SigScheme is only allowed for Unicode. In other words,
456
allowed only if the reading port is UTF-8. This limitation is intended to avoid
457
character identity problem between different character encodings. For example,
458
WAVE DASH and FULLWIDTH TILDE may be altered to another unexpectedly if the
459
source code is converted to/from non-Unicode Japanese encoding. Inhibiting
460
non-Unicode identifiers is the simplest way to avoid such problems.</p>
461
<p>To use Unicode identifiers, prepend following line to the source code. See also
462
"Ports" section to understand its mechanism.</p>
463
<div class="listingblock">
464
<div class="content">
465
<pre><tt># /usr/bin/env sscm -C UTF-8</tt></pre>
469
<div class="sectionbody">
473
Each port is associated with a single character encoding when it is open
477
<div class="listingblock">
478
<div class="content">
479
<pre><tt>sscm> (current-input-port)
480
#<iport mb UTF-8 file stdin></tt></pre>
485
The encoding of a port is not switched to another once it is open (NOTE: a C extension that operates on low level port can alter it)
490
The encoding of a port is determined by <tt>%%current-char-codec</tt> on its opening
494
<div class="listingblock">
495
<div class="content">
496
<pre><tt>$ sscm -C UTF-8
497
sscm> (%%current-char-codec)
500
sscm> (open-output-file "/tmp/sigscheme.tmp")
501
#<oport mb UTF-8 file /tmp/sigscheme.tmp>
503
sscm> (with-char-codec "ISO-8859-1"
504
(lambda () (open-output-file "/tmp/sigscheme.tmp")))
505
#<oport mb ISO-8859-1 file /tmp/sigscheme.tmp></tt></pre>
510
<tt>%%current-char-codec</tt> does not affect already open ports
514
<div class="listingblock">
515
<div class="content">
516
<pre><tt>sscm> (current-input-port)
517
#<iport mb UTF-8 file stdin>
519
sscm> (with-char-codec "ISO-8859-1"
521
(list (%%current-char-codec) (current-input-port))))
522
("ISO-8859-1" #<iport mb UTF-8 file stdin>)</tt></pre>
527
Characters output to a port is encoded to a byte stream according to the port's own character encoding regardless of <tt>%%current-char-codec</tt> value
531
<div class="listingblock">
532
<div class="content">
533
<pre><tt>$ sscm -C ISO-8859-1
534
sscm> (current-output-port)
535
#<oport mb ISO-8859-1 file stdout>
537
sscm> (with-char-codec "UTF-8" (lambda () (write #\x3042)))
538
Error: ScmMultibyteCharPort: invalid character</tt></pre>
540
<p>To specify per-file character encoding, prepend following line into the
541
file. The <tt>load</tt> procedure detects this line and temporarily switches
542
<tt>%%current-char-codec</tt> and the encoding of the reading port until read all
543
expressions from the file.</p>
544
<div class="listingblock">
545
<div class="content">
546
<pre><tt>#! /usr/bin/env sscm -C UTF-8</tt></pre>
550
<div id="footer-text">
551
Last updated 17-May-2008 12:45:34 JST
added
removed
sigscheme/doc/multibyte.html
