XML tools and big documents

[Tyler Baker]

Don Park wrote:

> >For my implementation, for ot.xml (a 4 meg document) only about 1-2 megs of
> RAM
> >is used to store the 4 meg file in RAM due to all Names being cached at the
> >parser level.  It also takes only 10-12 seconds with a P-120 running
> Symantec's
>
> My test results were from running on Atari 800 (just kidding <g>).  My test
> machine is Pentium-133 with JDK 1.1.6 with JIT enabled.  Building DOM is a
> slow process but there are intermediate forms I am investigating which cuts
> down DOM loading drastically.

Most of all of the hard work of building the DOM tree is already done by the
parser.  Right now with the current state of Java, the single most important
thing I have found that you can do to speed up building the DOM tree is to only
allocate memory for container structures like Arrays, Vectors, or other utility
Collection classes as needed.  In other words, an leaf-node should not have any
memory allocated for storing children.

As for the memory issue, I have thought about some sort of LZW compression of all
of the text in a document tree.  This would save a lot of memory, but may slow
down building the DOM tree a bit.  Any ideas on this?

> >JIT for JDK 1.2 b4 to build the entire DOM tree.  For spitting out the DOM
> tree
> >(and normalizing all the Text nodes) it takes about 15-20 seconds of which
> 5
> >seconds is spent normalizing text nodes and most of the rest of this time
> is
> >actually spent in a brute force search and replace method that scans all
> >character data and attribute values and replaces any occurrences of entity
> values
> >with entity names.  This can be very expensive but I know no other way
> around it.
>
> Why are you normalizing text nodes before writing them out?  Also, blindly
> replacing entity values with entity names is error prone.

In general I agree with this, and this sort of stuff should be done at the
application level.  Nevertheless, the programmer I feel should have a choice
whether to manually do this or let the formatter do all of the hard work.

As for the text nodes, they do not have to be normalized, just that in my tests I
accounted for this since many people will want to normalize the document tree to
make the output look pretty.

Tyler


xml-dev: A list for W3C XML Developers. To post, mailto:xml-dev at ic.ac.uk
Archived as: http://www.lists.ic.ac.uk/hypermail/xml-dev/
To (un)subscribe, mailto:majordomo at ic.ac.uk the following message;
(un)subscribe xml-dev
To subscribe to the digests, mailto:majordomo at ic.ac.uk the following message;
subscribe xml-dev-digest
List coordinator, Henry Rzepa (mailto:rzepa at ic.ac.uk)