\documentstyle[12pt]{article}
\begin{document}

\hyphenation{Com-pu-ter-Pro-cess-able}

\begin{center}

{\bf TEI MLW1}

{\small \bf Guidelines for Writing SGML DTDs and Instances of SGML
DTDs}

S.A. Mamrak, J. Barnes and M.J. Kaelbling\\
% revisions suggested 9 Feb 89 by C. M. Sperberg-McQueen
% Minor points include:
%        - "DTD's" changed to "DTDs"
%        - "dataset" changed to "document" to avoid confusion with the
%             MVS usage of "dataset" where most people say "file"
%        - occasional recasting for stylistic reasons, as marked
%

\today
\end{center}

\section{Introduction}

%  The Text Encoding Initiative (TEI) has decided to use SGML as the
%  basis for representing encoded documents.
The text encoding scheme formulated by the Text Encoding Initiative
(TEI) will use the syntax of ISO 8879, ``Standard Generalized Markup
Language'', (SGML) as the basis for representing encoded documents, with
the restrictions specified in this and later documents of the
Initiative.
%
%                                            SGML requires that Document
%  Type Definitions, DTD's, be specified for each type of document to be
%  encoded.  DTD's define a tag set, and the rules for ordering tags.
%  The rules are specified using an extended BNF notation.  Individual
%  documents, i.e., datasets, can then be marked up using these tag sets.
%  These individually marked up datasets are called instances of DTD's.
SGML specifies both (1) rules for marking up text elements (delimiters
to be used to signal markup tags, delimiters for ``entity references''
and so on) and (2) rules for defining the tag sets to be used in marking
up a documents and the hierarchical relationships of individual tags
(what tags, or more precisely what tag-delimited text elements, are
allowed or required to appear within what text elements).  The hierarchy
of text elements and their tags is defined by a Document Type
Definition, DTD.  Each marked up document is an instance of a DTD
(sometimes of more than one).

%    The responsibilities of the Syntax and Metalanguage Committee of
%    the TEI are (1) to determine if SGML is powerful enough to meet
%    the analysis needs of the TEI community, and (2) to ensure that
%    SGML DTD's and DTD instances defined in the project are amenable
%    to processing by computers.
The responsibilities of the Syntax and Metalanguage Committee of the TEI
are (1) to ensure that the scheme developed by the TEI is powerful
enough to meet the analysis needs of the text-research community, (2) to
ensure that the specific recommendations of the TEI guidelines conform
to SGML, and (3) to ensure that texts encoded according to the TEI
guidelines (and therefore their DTDs) are easily processable by
computer.
%                                 The nature of analyses to be
%    undertaken by the TEI community is currently under independent
%    evaluation by other TEI Committees.  Their findings will lay the
%    basis for determining the sufficiency of SGML.  It is the
%    purpose of this paper to make recommendations that will ensure
%    that the DTD's and their instances, as they are defined within
%    the TEI project, are amenable to processing by computers.
%
Other TEI Committees are responsible for defining concretely the
requirements for computer-assisted textual research.  Their work will
show whether SGML suffices to those requirements or must be somehow
extended or modified.  At present, no extensions are seen to be
necessary.  It is the purpose of this paper to make recommendations that
will ensure that the DTDs defined within the TEI project, and thus
documents which follow them, are readily amenable to processing by
computers.

\section{Processing SGML with Computers}

%  In this paper, guidelines will be suggested for defining SGML DTD's
%  and instances of DTD's.  As the grammar describing SGML has over 200
%  productions and many pages of explanation, elaboration and
%  clarification, it is inappropriate to present a detailed description
%  here.  Therefore, the interested reader is referred to the standard
%  itself for a detailed description of SGML.  The complete description,
%  however, is not necessary for an understanding of the guidelines given
%  below.
In this paper, guidelines will be suggested for defining SGML DTDs and
instances of DTDs.  No detailed description of SGML grammar can or need
be presented here. The interested reader is referred to the standard
itself or to the increasing volume of explanatory material for a
detailed description of the standard.  The complete description,
however, is not necessary for an understanding of the recommendations
given here.

%  \subsection{The Use of Existing Software Tools}
%
%  Since SGML is a language intended for computer-based systems, it is
%  reasonable that it should follow established conventions within the
%  realm of computer science.  By following accepted methods of notation
%  and structural configuration, languages can be processed by existing
%  tools following well-understood theories and techniques, thus offering
%  considerable savings to development efforts.  These savings are
%  realized by automatically performing tedious, error-prone calculations
%  correctly and by checking and clarifying the formal descriptions of
%  the languages.
%
%  Tokenizing and parsing are of major concern to computer science.
%  Tokenizing is the process of recognizing significant strings in a
%  language and parsing is the process of recognizing the language
%  itself.  There are several common techniques of tokenizing and
%  parsing, each with known properties and limitations.  Some of these
%  techniques are so well established that programs exist that can
%  automatically generate tokenizers and parsers for conforming language
%  descriptions.  The use of these tokenizer and parser generators goes a
%  long way to ensuring the rapid production of reliable, efficient
%  tokenizing and parsing software.  Furthermore, the theoretical
%  underpinnings of tokenizing and parsing serve to identify ambiguity
%  that must be removed from a language description.
%
%  \subsection{Some Implications of Using Computer-Processable SGML}
%
%  Two points must be emphasized as a result of our call for
%  machine-processable DTD's.  The first of these is that rendering
%  DTD's, and instances of DTD's, processable by computers may render
%  them less readable or less understandable by humans. This should not
%  be viewed as an obstacle to development by humans, however, since
%  human concerns can be taken into account by applying various {\em
%  filters} to the machine-processable form.  For example, a
%  user-interface for creating instances of DTD's may provide a
%  human-friendly view to the human user, using short or omitted tags,
%  say, but a machine-processable, rigorously marked-up view for
%  application software.  We highly recommend separating considerations
%  regarding readability or usability by machine and by humans in order
%  to control the complexity of the definition and analysis of DTD's and
%  their instances.
It should be emphasized that ensuring the easy machine-processability of
DTDs and their instances may render both DTDs and documents less
readable or less understandable by humans. This should not be viewed as
an obstacle to development by humans, however, since a more easily
readable view can be created by applying various {\em filters} to the
machine-processable form, and vice versa.
For example, a user-interface for creating instances of DTDs may provide
a human-friendly view to the human user, using short or omitted tags,
say, but a readily machine-processable, rigorously marked-up view for
application software.  Similarly, preparatory filters can transform
a file prepared using short references or omitted tags into a more fully
marked up form that is easier to process using tokenizers and parsers
generated automatically by commonly available software-development
tools.

We highly recommend separating considerations regarding readability or
usability by machine and by humans in order to control the complexity of
the definition and analysis of DTD's and their instances.
% This sentence added / proposed by msm ...
This separation may result in the definition of parallel DTDs for the
same classes of documents:  one intended to allow easily readable
encodings, and one designed to ensure easy software processing.  %
Alternatively, read:  [OR:  This separation may result in the definition
of DTDs designed to encourage easily readable documents together with
recommendations for transforming the document into a less easily read
but more easily processed form, before interchange or processing.]

%  The second point to be raised is that sufficient computer-science
%  expertise must be present in order to meet and maintain the goals of
%  machine-processability of DTD's.  The TEI must undertake to put such
%  expertise in place before commencement of the official definition of
%  TEI DTD's.
All
DTDs prepared in the course of the Initiative should be examined
specifically in the light of formal-language theory with an eye to
ensuring their tractability for processing.

\section{Rules for Document Markup}

%  Given that DTD's are defined according to the above guidelines,
%  certain style conventions should be imposed when creating instances of
%  DTD's to further ensure ease of machine processing.
%
%  \begin{itemize}
\subsection{Concrete Syntax}

% this graf added ... -msm
Except in special cases, the Concrete Reference Syntax of ISO 8879 should
be used for TEI encodings.  When character sets not conformant with
ISO 646 or ANSI X3.4 are used, corresponding graphic characters should
be used for delimiters whenever corresponding graphic characters exist.

\subsection{Attribute Values}

%  minor stylistic alterations in this graf ... -msm
When specifying attribute values in a document, use the style {\em
name=`value'}.  The {\em name} and {\em value indicator} (or {\em vi},
here the equal sign) must be explicitly written and the value of the
attribute must be enclosed in apostrophes.  If the value itself contains
apostrophes, they should be represented by entity references.  This rule
ensures an unambiguous interpretation of attributes and their respective
values, which can otherwise be problematic.

\subsection{White Space}

%  minor stylistic alterations in this graf ... -msm
When entering tags into the document, avoid extraneous {\em white space}
characters (tabs, carriage returns, line feeds, form feeds, and so on)
within the tag delimiters (the angle brackets).  This rule ensures that
no new characters are introduced into the document itself during
processing.

%  \end{itemize}

\section{Rules for Writing DTD's}

%  We have considerable previous experience with parsing \cite{kn:KAE88}
%  and translating \cite{kn:MAM87c} SGML DTD's.  In this section we
%  describe the guidelines that must be followed in the creation of SGML
%  DTD's if they are to be processable by existing software tools.  We
%  justify the guidelines only with minimal discussion and motivation.
%  For a detailed justification the reader should consult the sources
%  cited above or their authors.
%
%  \begin{itemize}

The following rules should serve to ensure that DTDs are readily
processable by existing software-development tools; they arise out of
previous experience parsing \cite{kn:KAE88} and translating
\cite{kn:MAM87c} SGML DTDs; for justification see the sources cited.

\subsection{Avoid Most Optional Features}

% Substantive changes to this graf ... -msm
%  \item Use only the markup prescribed for {\em minimal} SGML documents.
%  In particular, short references and the omit tag feature of SGML are
%  not used in these documents. The use of short references and omit tags
%  call for special-purpose processing in both the token-recognition and
%  parsing stages of evaluating an SGML DTD and make them unsuitable for
%  straight-forward processing by tokenizer and parser generators.
%  Filters can be used to generate and (re)specify omittable tags, once
%  the grammar is in place.  Similarly, a second, manual pass can be
%  executed to indicate acceptable tag abbreviation conventions.
DTDs intended to define the simplest machine-processable form of an
encoding should use only the {\em required} features of SGML and avoid
optional features wherever possible.  In particular, short references
and the OMITTAG feature should be avoided, since they call for special
processing by tokenizer and parser generators.

Optional features should also be avoided in DTDs which define a more
human-readable form of the encoding, since they are not guaranteed to
be present in all SGML software.  OMITTAG and SHORTREF form the major
exception to this rule.  They may be used, but should be used with care
to ensure that they do not allow constructs unintentionally.

%this graf suppressed ... -msm
%  \item Use attributes only when information must be specified outside of the
%  hierarchical structure of the DTD.  The tokenizing and parsing of
%  attributes is different from that of tags and a separate processing
%  mechanism must be employed when attributes are used in the DTD
%  specification.  Thus, they should be used only when absolutely
%  required.  Tags already provide a mechanism for specifying all
%  hierarchical structure in a document, and they should be used for this
%  purpose.  If information must be specified that is not within the
%  hierarchical structure  (e.g., the relationship between two parts of
%  the verb in some German sentences), then attributes may be used.

\subsection{Inclusion and Exclusion Exceptions}
%  minor stylistic changes ... -msm
Avoid inclusion and exclusion exceptions in the DTD grammar.  For parser
generators, such exceptions must be tediously specified on a
production-by-production basis, although a tool might be implemented to
do this automatically.  More importantly, however, experience indicates
that when they are used, the grammar often grows inordinately large and
unintended constructs are likely to occur.

\subsection{Modularity}

Use principles of modular design to guide the specification of
the DTD. Human comprehension of these grammars is limited and as they
become complex unintended constructs may arise. In general, it is
better to employ many simple subgrammars that capture the unique
content of some content than to specify a few, large subgrammars that
become unwieldy and go beyond human comprehension.  Specific
guidelines in this case include:

\begin{itemize}

% numbers changed -- OK now?  ... -msm
\item Use only 40-50 productions in a grammar.  If
there is a need for a grammar to become much larger than this,
subgrammars should likely be used.

\item  The tree represented by the grammar should be only 9-10 levels
deep, and probably not even so deep.  Again, consider subgrammars if
depth is much beyond this.

\item Define only simple content models (i.e., subgrammars) and use
them with care.

\end{itemize}

\subsection{Multiple Hierarchical Structures}
%  Changes to style and substance in this graf ... -msm
If more than one hierarchical structure is to be tagged in a document,
each hierarchy should be independently specified in a separate DTD to
begin with.  Trying to combine multiple hierarchies in a single DTD from
the beginning will lead to unnecessary complexity.  Several approaches,
including use of the {\em concur} feature of SGML, can be used to handle
such cases; more detailed recommendations will follow further study of
the issue.

%  \end{itemize}

\bibliography{/n/fruit/0/mamrak/publications/ieee-tse/cham}
\bibliographystyle{plain}

\end{document}
