XML and special Characters : unicode v3.0 ?

[MURATA Makoto]

John Cowan writes:
>Tim Bray writes:
>> In practice,
>> I've never actually seen anything outside of the BMP, but the
>> experts agree they're showing up real soon now.
>
>Not until Unicode 4.0, unless someone wants to use the private-use
>planes 15 and 16.

It is my understanding that Unicode 3.0 will have many ideographic 
characters which are outside of the BMP.

>John Cowan writes:
>Tim Bray writes:
>> So... it seems that in UTF8,
>> a ZWNBSP as first character in the file isn't a data character.
>
>Can you quote chapter and verse for this, either Unicode or 10646?
>The latter spec tells you that the sequence EF BB BF may be used as
>a *signature* at the beginning of UTF-8 data (since it is unlikely
>to occur in any other kind), but does not IMHO imply that the
>sequence is removable or doesn't represent a real ZWNBSP.

Attached is quoted from A2 of N1396 ISO/IEC 10646-1 Corrigendum 
no. 2 (First draft - revised to 30 April 1996), which was (is?) available 
at http://osiris.dkuug.dk/JTC1/SC2/WG2/docs/N1396.doc

The para most relevant to your question is:
>An application receiving data may either use these signatures to
>identify the coded representation form, or may ignore them and treat
>FEFF as the ZERO WIDTH NO-BREAK SPACE character.

How do you interpret this "or"?   One could argue that when EF BB BF 
is recognized as a signature, it is not treated as the ZWNS.  Unfortunately, 
every description about the BOM (even for UCS-2 or UTF-16) is unclear 
and subject to different interpretations, as I see it.

Cheers,


Makoto
 
Fuji Xerox Information Systems
 
Tel: +81-44-812-7230   Fax: +81-44-812-7231
E-mail: murata at apsdc.ksp.fujixerox.co.jp
---------------------------------------------------------

Annex F
(informative)
The use of "signatures" to identify UCS


 This annex describes a convention for the identification of features
of the UCS, by the use of "signatures" within data streams of coded
characters. The convention makes use of the character ZERO WIDTH
NO-BREAK SPACE, and is applied by a certain class of applications.

When this convention is used, a signature at the beginning of a stream
of coded characters indicates that the characters following are
encoded in the UCS-2 or UCS-4 coded representation, and indicates the
ordering of the octets within the coded representation of each
character (see 6.3). It is typical of the class of applications
mentioned above, that some make use of the signatures when receiving
data, while others do not. The signatures are therefore designed in a
way that makes it easy to ignore them.　In this convention, the ZERO
WIDTH NO-BREAK SPACE character has the following significance when it
is present at the beginning　of a stream of coded characters:

UCS-2 signature: FEFF

UCS-4 signature: 0000 FEFF

UTF-8 signature: EF BB BF

UTF-16 signature: FEFF

An application receiving data may either use these signatures to
identify the coded representation form, or may ignore them and treat
FEFF as the ZERO WIDTH NO-BREAK SPACE character.

If an application which uses one of these signatures recognises its
coded representation in reverse sequence (e.g. hexadecimal FFFE), the
application can identify that the coded representations of the
following characters use the opposite octet sequence to the sequence
expected, and may take the necessary action to recognise the
characters correctly.

NOTE - The hexadecimal value FFFE does not correspond to any coded
character within ISO/IEC 10646.

Makoto
 
Fuji Xerox Information Systems
 
Tel: +81-44-812-7230   Fax: +81-44-812-7231
E-mail: murata at apsdc.ksp.fujixerox.co.jp

xml-dev: A list for W3C XML Developers. To post, mailto:xml-dev at ic.ac.uk
Archived as: http://www.lists.ic.ac.uk/hypermail/xml-dev/ and on CD-ROM/ISBN 981-02-3594-1
To (un)subscribe, mailto:majordomo at ic.ac.uk the following message;
(un)subscribe xml-dev
To subscribe to the digests, mailto:majordomo at ic.ac.uk the following message;
subscribe xml-dev-digest
List coordinator, Henry Rzepa (mailto:rzepa at ic.ac.uk)