Binary data

[sliekens]

At this time, the ITextScanner interface does not provide APIs for binary data.

The workarounds are:

1. Use OctetLexer to read any number of bytes as instances of Element
2. Read directly from the underlying stream

The first workaround is nasty for a number of reasons:

• Overhead: the OctetLexer class creates an instance of the Element class for every byte.
• Conversion: the OctetLexer class converts each byte to System.String; you have to convert it back to a byte using ITextScanner.Encoding.

The second workaround isn't much better:

• You already have to have a reference to the underlying stream to make it work
• Reading from the stream does not update the scanner position

Suggested fix: copy the Read and ReadByte + async variants from System.IO.Stream to the ITextScanner interface. Implement these methods in a way that updates the scanner's internal state.

[sliekens]

The suggested fix has its own issues...

• The scanner's internal state (next character, current position) is represented by characters, not bytes. This is very problematic for variable-width character encodings.
• Simply adding Read methods to the ITextScanner interface does not make it compatible with APIs that operate on a System.IO.Stream.

Suggested alternative fix: add a property ITextScanner.BaseStream. The implementation of that property should return a wrapper around the underlying stream. The wrapper should implement the following behavior:

• Write()/Seek() should throw NotSupportedException
• Read() & friends should notify the scanner of what was read (using callbacks or events)
• The wrapper's Dispose() method should remove its own references to the underlying stream, but not close it.
  • This is to stop users from disposing the underlying stream through its wrapper. The underlying stream should only be closed and garbage-collected when the scanner object is closed. However, lingering references to disposed wrappers could prevent the stream from being garbage-collected unless those references are removed when the wrapper is disposed.
  • Actually, this looks like it should be a weak reference

[sliekens]

More ideas...

Instead of the BaseStream property, add a method ITextscanner.ReadRaw(Action<Stream>)

The implementation of this method is responsible for managing the lifetime of objects

void ReadRaw(Action<Stream> callback)
{
    using (Stream wrapper = new WrapStream(inputstream, this))
    {
        callback(wrapper);
    }
}

This way, the scanner object has full control over all objects: the wrapper stream becomes unusable after the callback method returns.

TODO: figure out what pattern that the WrapStream should use for notifying the scanner object when it is read.

[sliekens]

Since ReadRaw(callback) lets the caller read data out of context, which may be binary data or character data in a different encoding, is there any meaningful way to maintain the scanner's internal state?

I think that a ReadRaw action should always force the scanner back to its pre-initialized state.

• Position -1
• NextCharacter unknown

Discarding internal state is the only way to prevent integer overflow when the number of raw bytes read is greater than int.MaxValue (2GB).

[sliekens]

I added two members to ITextScanner

1. ITextScanner.Reset()
2. ITextScanner.BaseStream

The Reset() method sets the internal state to the pre-initialized state and releases any internal buffers that it may hold. This method should always be safe to call in between reads.

The BaseStream property returns a direct reference to the underlying stream. Callers should take care not to dispose this stream, and to call Reset() before attempting to read from this stream.