XML tools and big documents

[Nigel Kerr]

Quoth David Megginson <david at megginson.com>:

> XML is for interchange -- for simple applications you can use it for
> storage (as I do on my notebook), but for larger, multi-user
> applications, you probably want to put it into some kind of
> specialised storage, if only for the sake of revision and access
> control.

My interest in descriptive markup in general is in describing large,
relatively static (meaning they don't get revised, usually), text
collections: the many fine TEI or EAD encoding projects, for instance,
the various literature collections one can get from Chadwyck-Healey,
and the like (our shop here takes such things and indexes them with
some version or other of OpenText for searching and structured
retrieval).  These are all described by SGML DTD's.

I've seen discussion and work on making an XML DTD to correspond as
closely as possible to the TEI, similarly with EAD.  The kinds of
documents these two DTD's can describe can be arbitrarily large (the
average EAD finding aid is relatively small, but we have a couple
pushing several megabytes).  Are there then folks interested in XML
for things other than interchange?  Authoring, certainly, but also
storage and retrieval of large text collections?

To this end, I have been (in such spare time as i have) tinkering
about with Mr. Clark's XP API (com.jclark.xml.tok, mostly) to write an
application that will allow me to attach the logical element structure
to offsets in the storage entity, so that I can consider the logical
structure's relationship to points in the text without reparsing the
document.  I want to be able to ask questions like:

	"what's the most immediate containing element of offset X in
	file Y?"

	"traverse up the logical structure from offset X until a DIV
	element with a HEAD is found, and return me the offsets of
	that HEAD"

Exact expression language is, uh, gee.  These are the kinds of
questions we could ask with "some XML query language", but if i have a
gigabyte or so of variously-structured English text marked up this
way, i really don't want to have to parse the document entity just to
answer these kinds of simple questions.  This is a weak specification
of what I'm trying to do, i realize.  (this all largely because i am
disatisfied with the limited information of the logical tree that
OpenText's sgmlrgnXX gives... ).

Anyone else here interested in these kinds of problems, and using XML
tools on them?


Nigel Kerr				              nigelk at umich.edu
Digital Library Production Services         http://www.umdl.umich.edu/
University of Michigan

xml-dev: A list for W3C XML Developers. To post, mailto:xml-dev at ic.ac.uk
Archived as: http://www.lists.ic.ac.uk/hypermail/xml-dev/
To (un)subscribe, mailto:majordomo at ic.ac.uk the following message;
(un)subscribe xml-dev
To subscribe to the digests, mailto:majordomo at ic.ac.uk the following message;
subscribe xml-dev-digest
List coordinator, Henry Rzepa (mailto:rzepa at ic.ac.uk)