A Simple Thought

[Stephen D. Williams]

roddey at us.ibm.com wrote:

> >Ahh, there's the trick.  I believe I have most of a design for an data
> structure
> >that is fast in memory yet is 'flat' and can have its chunks just written
> out or
> >read in at any point.  It builds on some very old ideas I came up with for
> a
> >language I designed.  When viewed as an interchange format, it may not be
> the
> >most optimal space wise (although it should be better than XML text) but
> trades
> >a small amount of space for nearly zero processing overhead.  There will
> >probably also be a procedure for 'compacting' an object for storage into a
> >database or sending over a slow link vs. the 'fast' format usable between
> >servers in a cluster.
> >
>
> I think though that this would only hold up as long as you are looking at
> XML data as a read-only data source. Once you started doing significant
> editing of the data, having a flat structure like that would be more of a
> hinderance than a help, would it not? What if I have a 10MB flat buffer and
> want to add another child to the second element? This kind of gets into the
> quandry that you've nailed in one nail, but now its even harder to nail a
> whole raft of others as well as with the more general purpose mechanisms.
>
> I dunno, if I were thinking along these lines, to keep it reasonably
> portable, I'd look at the binary format as a fast serialization mechanism
> and at least create native language objects for each one. By the time you
> put enough stream format markers and whatnot into the stream to know where
> things are, and interpret those during runtime, it might be just as fast to
> pay the cost for creating a much more flexible, native object format for in
> memory manipulation.

I have a way around the modification issue.  It's a data structure I call 'elastic memory'.
It's really the main reason that I'm going to have to start mostly from scratch.
I AM trying to hit a number of nails at once and it won't be easy and I'm not sure I can make
it perfect, however I believe I can get close.  I'm only worrying about Java at the moment
with some allowances for certain restrictions that come into play and typical usage in network
protocols.

There are a number of situations where serialization just doesn't cut it.  As I mention,
imagine serializing/deserializing on every method call in a program.

I designed some of these mechanisms MANY years ago (about 8 I think) while designing a
language after I'd already built a language based on Postscript syntax for a project.  In
testing the first language, I learned the horrors of 'malloc storms' that happen when you
follow a typical design paradigm.  My system which allowed a complex application to be
represented by meta data would do about 25000 mallocs in a standard run through the app.  A
Java web server app for a very complex app  I just completed with a team does about 150,000
object creations (measured by forcing a garbage collection) in one run through the app.  It
works amazingly well, but still blows most of it's processing for things that could be
avoided.

The cool thing is that I found a way to implement it in Java.

Thanks for sparring with me! ;-)
sdw

> xml-dev: A list for W3C XML Developers. To post, mailto:xml-dev at ic.ac.uk
> Archived as: http://www.lists.ic.ac.uk/hypermail/xml-dev/ and on CD-ROM/ISBN 981-02-3594-1
> To (un)subscribe, mailto:majordomo at ic.ac.uk the following message;
> (un)subscribe xml-dev
> To subscribe to the digests, mailto:majordomo at ic.ac.uk the following message;
> subscribe xml-dev-digest
> List coordinator, Henry Rzepa (mailto:rzepa at ic.ac.uk)

--
OptimaLogic - Finding Optimal Solutions     Web/Crypto/OO/Unix/Comm/Video/DBMS
sdw at lig.net   Stephen D. Williams  Senior Consultant/Architect   http://sdw.st
43392 Wayside Cir,Ashburn,VA 20147-4622 703-724-0118W 703-995-0407Fax 5Jan1999



xml-dev: A list for W3C XML Developers. To post, mailto:xml-dev at ic.ac.uk
Archived as: http://www.lists.ic.ac.uk/hypermail/xml-dev/ and on CD-ROM/ISBN 981-02-3594-1
To (un)subscribe, mailto:majordomo at ic.ac.uk the following message;
(un)subscribe xml-dev
To subscribe to the digests, mailto:majordomo at ic.ac.uk the following message;
subscribe xml-dev-digest
List coordinator, Henry Rzepa (mailto:rzepa at ic.ac.uk)