All document encoding has to do with representing one thing by
another in an agreed and systematic way. Applied to the smallest
distinctive units in any given writing system, which for the
moment we may loosely call
When the first methods of representing text for storage or
transmission by machines were devised, long before the
development of computers, the overriding aim was to identify the
smallest set of symbols needed to convey the essential semantic
content, and to encode that symbol set in the most economical
way that the storage or transmission media allowed. The initial
outcome were systems that encoded only such content as could be
expressed in uppercase letters in the Latin script, plus a few
punctuation marks and some
For many years after the invention of computers, the way they
represented text continued to be constrained by the imperative
to use expensive resources with maximal efficiency. Even when
storage and processing costs began their dramatic fall, the
Anglo-centric outlook of most hardware designers and software
engineers hampered initiatives to devise a more generous and
flexible model for text representation. The wish to retain
compatibility with
Unfortunately, in the early years there was little or no co-ordination among either the national standards bodies or the manufacturers concerned, so that although commercial necessity dictated that these various local standards were all compatible with the representation of US-American English, they were not straightforwardly compatible with one another. Even within Japan itself there emerged a number of mutually incompatible systems, thanks to a mixture of commercial rivalry, disagreements about how best to manage certain intractable problems, and the fact that such pioneering work inevitably involved some false starts, leading to incompatibilities even between successive products of the same bodies. Roughly at the same time, and for similar reasons, multiple and incompatible ways of representing languages that use Cyrillic scripts were devised, along with methods of encoding ancient writing systems which inevitably could not aim for compatibility with other writing systems apart from basic Latin script. Many of the earliest projects that fed into the TEI were shaped in this developmental phase of the computerized representation of texts, and it was also the context in which SGML was devised and finalized.
SGML had of necessity to offer ways of coping with multiple writing systems in multiple representations; or rather, it provided a framework within which SGML-compliant applications capable of handling such multiple representations might be developed by those with sufficient financial and personnel resources (such as are seldom found in academia). Earlier editions of these Guidelines offered advice on character set and writing system issues addressed to the condition of those for whom SGML was the only feasible option. That advice is here substantially altered because of two closely-related developments: the availability of the ISO/Unicode character set as an international standard, and the emergence of XML and related technologies which are committed to the theory and practice of character representation which Unicode embodies.
Before the significance of Unicode and the implications of the association between XML and Unicode can be adequately explained, it is necessary to clarify some key concepts and attempt to establish an adequately precise terminology for them.
Examples of the latin
The word
In most scholarly encoding projects, the accurate recording of the abstract characters which make up the text is of prime importance, because it is the essential prerequisite of digitizing and processing the document without semantic loss. In many cases (though there are important exceptions, to be touched on shortly) it may not be necessary to encode the specific glyphs used to render those abstract characters in the original document. An encoding that faithfully registers the abstract characters of a document allows us to search and analyse our document's content, language, and structure, and to access its full semantics. That same encoding, however, may not contain sufficient information to allow an exact visual representation of the glyphs in the source text or manuscript to be recreated.
The importance of this distinction between information content
and its visual representation is not always immediately apparent
to people unused to the specific complexities of text handling
by machine. Such users tend to ask first what (in order of
conceptual priority) should actually be their very last
question: how do I get a physical image that looks like
character x in my source document to appear on to the screen or
the output page? Their first question should in fact be: how can
I get an abstract representation of character x into my encoded
document in a way that will be universally and unambiguously
identifiable, no matter what it happens to look like in printout
or on any particular display? And occasionally the response they
receive as a result of their misguided initial question is a
custom
That said, there will certainly be documents or projects where it is a matter of scholarly significance that the compositor or scribe chose to represent a given abstract character using one particular glyph or set of strokes rather than a semantically-equivalent but visually distinct alternative, and in that case the specific appearance of the form will have to be encoded in one way or another. But that encoding need not (and in most cases will not) involve a notation that visually resembles the original, any more than italicized text in an original document will be represented by the use of italic characters in the encoded version.
A collection of the abstract characters needed to represent
documents in a given writing system is known as a
It is now possible to use this terminology to
say what Unicode is: it is a coded character set, devised and
actively maintained by an international public body, where each
abstract character is identified by a unique name and assigned a
distinctive code point.
The distinction between abstract characters and glyphs can be crucial when devising an encoding scheme. When performing searches, text retrieval, or creating concordances, users of electronic text will expect the system to recognize and treat different glyphs as instances of the same character; but when perusing the text itself they may well expect to see glyph variants preserved and rendered. When encoding a pre-existing text, the encoder should determine whether a particular letter or symbol is a character or a glyphic variant. The Unicode Consortium and an ISO work group (ISO/IEC JTC1 SC2/WG2) have developed a detailed model of the relationship between characters and glyphs. This model, presented in Unicode Technical Report 17: Character Encoding Model, is the underpinning of much standards work since, including the current chapter.
The model makes explicit the distinction between two different
properties of the components of written language:
When searching for information, a system generally operates
on the content aspects of characters, with little or no
attention to their appearance. A layout or formatting process,
on the other hand, must of necessity be concerned with the exact
appearance of characters. Of course, some operations
(hyphenation for example) require attention to both kinds of
feature, but in general the kind of text encoding described in
these Guidelines tends to focus on content rather than
appearance (see further
An encoder wishing to record information about which glyphs
are present in a given document may do so at either or both of
two levels:
The encoding practice adopted may be guided by, among other things, an assessment of the most frequent uses to which the encoded text will be put. For example, if recognition of identical characters represented by a variety of glyphs is the main priority, it may be advisable to represent the glyph variations at markup level, so that the character value can be immediately exposed to the indexing and retrieval software. Plainly, an encoding project will need to consider such issues carefully and document the outcome of their deliberations in their TEI customization file (or other local encoding documentation) to ensure encoding consistency. Using Unicode code points to represent glyph information requires that such choices be documented in the TEI header. Such documentation cannot of itself guarantee proper display of the desired glyph but at least makes the intention of the encoder discoverable.
At present the Unicode Standard does not offer detailed
specifications for the encoding of glyph variations. These
Guidelines do give some recommendations; some discussion of
related matters is given in
The entry of characters was much more complicated before the near-universal
adoption of Unicode, for which there are where
D is an integer representing the code point of the character in
base 10, or , where H is the code point in
hexadecimal notation. Every XML processor is capable of recognising NCRs and
replacing them with the required code point value without needing access to
any additional data. The disadvantage of NCRs as a means of entering,
representing and proofing character data is that most human beings find them
anything but
The rendering of the encoded text is a complicated process that depends largely on the purpose, external requirements, local equipment and so forth, it is thus outside the scope of coverage for these Guidelines.
It might nevertheless be helpful to put some of the terminology used for the rendering process in the context of the discussion of this chapter. As was mentioned above, Unicode encodes abstract characters, not specific glyphs. For any process that makes characters visible, however, concrete, specifically designed glyph shapes have to be used. For a printing process, for example, these shapes describe exactly at which point ink has to be put on the paper and which areas have to be left blank. If we want to print a character from the Latin script, besides the selection of the overall glyph shape, this process also requires that a specific weight and size of the font has been selected, and to what degree the shape should be slanted. Beyond individual characters, the overall typesetting process also follows specific rules for calculating the distance between characters, for determining how much whitespace occurs between any two words, and how long each line should be (and thus at which points a new line begins), and so forth.
If we concern ourselves only with the rendering process of the characters themselves, leaving out all these other parameters, we will realize that of all the information required for this process, only a small amount will be drawn from the encoded text itself. This information is the code point used to encode the character in the document. With this information, the font selected for printing will be queried to provide a glyph shape for this character. Some modern font formats (e.g. OpenType) implement a sophisticated mapping from a code point to the glyph selected, which might take into account surrounding characters (to create ligatures where necessary) and the language or even area this character is printed for to accommodate different typesetting traditions and differences in the usage of glyphs.
A TEI document might provide some of the information that is
required for this process, for example by identifying the
linguistic context with the
XML was designed with Unicode in mind as its means of representing
abstract characters. It is possible to use other character encoding
schemes, but in general they are best avoided, as you run the risk
of encountering compatibility issues with different XML processors,
as well as potential difficulties with rendering their output. We
recommend using the
The principles of Unicode are judiciously tempered with
pragmatism. This means, among other things, that the actual
repertoire of characters which the standard encodes, especially
those parts dating from its earlier days, include a number of
items which on a strict interpretation of the Unicode
Consortium's theoretical approach should not have been regarded
as abstract characters in their own right. Some of these
characters are grouped together into a
code-point regions assigned to s
and t
or f
and i
in Latin
scripts, whether produced by a scribal practice of not lifting
the pen between strokes or dictated by the aesthetics of a type
design) are representational features with no added semantic
value beyond that of the two letters they unite (though for
historians of typography their presence and form in a given
edition may be of scholarly significance). However, by the time
the Unicode standard was first being debated, it had become
common practice to include single glyphs representing the more
common ligatures in the repertoires of some typesetting devices
and high-end printers, and for the coded character sets built
into those devices to use a single code point for such glyphs,
even though they represent two distinct abstract characters. So
as to increase the acceptance of Unicode among the makers and
users of such devices, it was agreed that some such
pseudo-characters should be incorporated into the standard as compatibility characters.
Nevertheless, if a project requires the presence of such
ligatured forms to be encoded, this should normally be done via
markup, not by the use of a compatibility character. That way,
the presence of the ligature can still be identified (and, if
desired, rendered visually) where appropriate, but indexing and
retrieval software will treat the code points in the document as
a simple sequential occurrence of the two constituent characters
concerned and so correctly align their semantics with
non-ligatured equivalents. Such ligatures should not be confused
with digraphs (usually) indicating diphthongs, as in the French
word "cœur". A digraph is an atomic orthographic unit
representing an abstract character in its own right, not purely an amalgamation
of glyphs, and indexing and retrieval software will need to
treat it as such. Where a digraph occurs in a source text, it
should normally be encoded using the appropriate code point for
the single abstract character which it represents.
The treatment of characters with
diacritical marks within Unicode shows a similar combination of
rigour and pragmatism. It is obvious enough that it would be
feasible to represent many characters with diacritical marks in
Latin and some other scripts by a sequence of code points, where
one code point designated the base character and the remainder
represented one or more diacritical marks that were to be
combined with the base character to produce an appropriate
glyphic rendering of the abstract character concerned. From its
earliest phase, the Unicode Consortium espoused this view in
theory but was prepared in practice to compromise by assigning
single code points to
It is important that every Unicode-based project should agree on, consistently implement, and fully document a comprehensive and coherent normalization practice. As well as ensuring data integrity within a given project, a consistently implemented and properly documented normalization policy is essential for successful document interchange. While different input methods may themselves differ in what normalization form they use, any programming language that implements Unicode will provide mechanisms for converting between normalization forms, so it is easy in practice to ensure that all documents in a project are in a consistent form, even if different methods are used to enter data.
In addition to the Universal Character Set itself, the
Unicode Consortium maintains a database of additional character
semantics
In addition to the printed documentation and lists made
available by the Unicode consortium, the information it contains
may also be accessed by a number of search systems over the Web
(e.g.
In theory it should not be necessary for encoders to have any
knowledge of the various ways in which Unicode code points can
be represented internally within a document or in the memory of
a processing system, but experience shows that problems
frequently arise in this area because of mistaken practice or
defective software, and in order to recognize the resulting
symptoms and correct their causes an outline knowledge of
certain aspects of Unicode internal representation is desirable.
There are three encodings of Unicode available for use: UTF-8, which
uses 1–4 bytes per character, UTF-16, which uses 2–4, and UTF-32,
which uses 4 bytes per character. Current practice for documents to
be transmitted via the Web recommends only UTF-8.
The code points assigned by Unicode 3.0 and later are notionally 32-bit integers, and the most straightforward way to represent each such integer in computer storage would be to use 4 eight-bit bytes. However, many of the code points for characters most commonly used in Latin scripts can be represented in one byte only and the vast majority of the remainder which are in common use (including those assigned from the most frequently used PUA range) can be expressed in two bytes alone. This accounts for the use of UTF-8 and UTF-16 and their special place in the XML standard. UTF-8 and UTF-16 are ways of representing 32-bit code points in an economical way.
UTF-8 is a variable length encoding: the more significant bits there are in the underlying code point (or in everyday terminology the bigger the number used to represent the character), the more bytes UTF-8 uses to encode it. What makes UTF-8 particularly attractive for representing Latin scripts, explaining its status as the default encoding in XML documents, is that all code points that can be expressed in seven or fewer bits (the 127 values in the original ASCII character set) are also encoded as the same seven or fewer bits (and therefore in a single byte) in UTF-8. That is why a document which is actually encoded in pure 7-bit ASCII can be fed to an XML processor without alteration and without its encoding being explicitly declared: the processor will regard it as being in the UTF-8 representation of Unicode and be able to handle it correctly on that basis.
However, even within the domain of Latin-based scripts, some
projects have documents which use characters from 8 bit
extensions to ASCII, e.g. those in the ISO-8859-n series of
encodings, and the way characters which under ISO-8859-n use
all eight bits are encoded in UTF-8 is significantly different,
giving rise to puzzling errors. Abstract characters that have a
transformed (hence the T in
UTF) into a sequence of two different numbers. Now as a
side-effect of the way such UTF-8 sequences are derived from
the underlying code-point value, many of the single-byte
eight-bit values employed in ISO-8859-n encodings are illegal
in UTF-8.
This complicated situation has a simple consequence which can
cause great bewilderment. XML processors will effortlessly
handle character data in pure 7-bit ASCII without that encoding
needing to be declared to the parser, and will similarly accept
documents encoded in an undeclared ISO-8859-n encoding if they
happen to use no characters outside the strict ASCII subset of
the ISO character sets; but the parse will immediately fail if
an eight-bit character from an ISO-8859-n set is encountered in
the input stream, unless the document's encoding has been
explicitly and correctly declared. Explicitly declaring the
encoding ought to solve the problem, and if the file is
correctly encoded throughout, it will do so. But projects dealing
with documents of sufficient age may find that they have to deal with some files encoded
in UTF-8 along with others in, say, ISO-8859-1. Such encoding
differences may go unnoticed, especially if the proportion of
characters where the internal encodings are distinguishable is
relatively small (for example in a long English text with a
smattering of French words). These types of error may or may not
manifest in actual processing errors, and may only become visible
as
In projects that routinely handle documents in non-Latin
scripts, everyone is well aware of the need to ensure correct
and consistent encoding, so in such places mixed encoding
problems seldom arise, and when they do are readily identified
and remedied. Real confusion tends to arise, however, in
projects which have a low awareness of the issues because they
employ predominantly unaccented Latin characters, with only
thinly-distributed instances of accented letters, or other