Streams, protocols, documents and fragments

[David Megginson]

Tom Harding writes:

 > David Megginson wrote:
 > 
 > > More importantly, you don't want to have to parse an entire document
 > > just to find out where it ends because that forces your system into
 > > linear processing -- on a busy server, it is absolutely necessary to
 > > be to isolate the documents/packets quickly and pass them off to
 > > separate threads (or even separate boxes) for parsing and processing.
 > 
 > Good point.  I have been implicitly assuming this as a cost of
 > moving the parsing function into the network infrastructure.
 > However, a general-purpose endpoint implementation would have a
 > hard time parallelizing in the way you describe because of possible
 > inter-document dependencies in the application protocol.  It has to
 > deliver the documents to the next layer in the order in which they
 > were sent.  If parallelism is explicitly needed then a solution is
 > to create multiple connections.

It's not necessarily the raw XML that will be delivered to the
application, however -- it could be that the XML will be preprocessed
to populate an object tree, create a 3D graphic, etc., and that kind
of processing can easily be done in parallel.  Imagine this payload:

Packet 1 of 4: RDF metadata description of payload
Packet 2 of 4: XML text for a news story on Amazon.com
Packet 3 of 4: XML vector-graphic format for a chart of Amazon's
               earnings
Packet 4 of 4: Base 64-encoded graphic of Amazon's logo

As soon as I receive packet #3, for example, I can hand it off to a
separate process for rendering, even though I am not yet ready to pass 
the rendered version to the application until the whole payload is
received.  I can prepare a set of SQL statements to execute to add the 
information in #1 to my database, I can convert #2 to HTML or PDF, and 
I can convert #4 to a PNG, all in parallel without any
interdependencies.  If I have to parse all of #1 before I can start on 
#2, my system will be much less efficient.

In other words, if the general-purpose system had some kind of typing
information available, it could do many types of generic processing
before worrying about dependencies.


All the best,


David

-- 
David Megginson                 david at megginson.com
           http://www.megginson.com/

xml-dev: A list for W3C XML Developers. To post, mailto:xml-dev at ic.ac.uk
Archived as: http://www.lists.ic.ac.uk/hypermail/xml-dev/ and on CD-ROM/ISBN 981-02-3594-1
To (un)subscribe, mailto:majordomo at ic.ac.uk the following message;
(un)subscribe xml-dev
To subscribe to the digests, mailto:majordomo at ic.ac.uk the following message;
subscribe xml-dev-digest
List coordinator, Henry Rzepa (mailto:rzepa at ic.ac.uk)