<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
          "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd">
<book>
<bookinfo>
  <title>The IRON Data Model Specification</title>
  <author><firstname>Alaric</firstname><surname>Snell-Pym</surname></author>
  <copyright><year>2007</year><holder>Warhead.org.uk Ltd</holder></copyright>
</bookinfo>
<preface><title>Foreword</title>

  <!-- FIXME: This is now a bit outdated - see intro/iron.wiki -->

<para>IRON is a data model for values. Although I'm still deciding how the mutable data structures like queues fit into things (specifications of them are definitely needed for TUNGSTEN, but whether they count as part of IRON or not is something I'm still debating), I think I may have settled on a basic syntax for written values.</para>

<para>Now, the key requirement here is that IRON is, in the manner of S-expressions, usable to express just about anything - from source code to boring data. Creating a written data syntax that's pleasant enough to use day in and day out is quite a challenge. s-expressions come pretty close, but are deficient in a few areas. YAML is pretty good, but I wouldn't want to write source code in it.</para>

<para>The main thing I'm adding over s-expressions is Smalltalk-like syntax, which I will explain in detail below.</para>

</preface>
<chapter><title>Textual form</title>

<para>Herein I define representations of IRON values in terms of sequences of characters. How those characters are encoded into a binary stream is another matter entirely.</para>

<sect1><title>Atoms</title>

<sect2><title>Integers</title>
<para>IRON integers can be anything from negative to positive infinity, subject only to the constraints of implementation. An implementation must be able to support at least what you'd get from 32 two's-complement signed bits, but should be able to support arbitrary precision integers, if memory permits.</para>

<para>There are several representations available:</para>

<itemizedlist>
  <listitem><para>A sequence of decimal digits, with an optional leading <literal>-</literal> symbol, represents an integer in base 10</para></listitem>
  <listitem><para>The sequence <literal>0x</literal> followed by a sequence of hexadecimal digits represents an integer in base 16. Negative integers can be represented with a leading <literal>-0x</literal>, not <literal>0x-</literal>.</para></listitem>
  <listitem><para>The sequence <literal>0b</literal> followed by a sequence of binary digits represents an integer in binary. Again, negative binary numbers are written starting with <literal>-0b</literal> not <literal>0b-</literal>.</para></listitem>
</itemizedlist>

</sect2>

<sect2><title>Rational numbers</title>
<para>Rationals can be written in decimal, hexadecimal, or binary, as per integers; they distinguish themselves from integers with the presence of a <literal>/</literal> at some point, dividing the number into an integer numerator at the start and a natural (positive integer) denominator at the end. A base prefix may occur at the start of the rational value, and applies to both numerator and denominator; the denominator may not have a base prefix if its own. For example, <literal>5/2</literal>, <literal>0x5/2</literal> and <literal>0b101/10</literal> all represent the same number.</para>
</sect2>

<sect2><title>Floating point numbers</title>
<para>Floats can be written in decimal, hexadecimal, or binary, as per integers; they distinguish themselves from integers with the presence of a <literal>.</literal> at some point; they may also optionally have an exponent appended, of the form <literal>e</literal> (FIXME: How to disambiguate from a hexadecimal e digit?) followed by an integer in the same base as the entire float (eg, <literal>0x</literal> and <literal>0b</literal> prefixes are forbidden, and the base is inhereted from the float's prefix, if any) with optional leading <literal>-</literal>. For example, <literal>2.5</literal>, <literal>.25e1</literal>, <literal>0b.101e10</literal>, and <literal>0x2.8</literal> all represent the same number.</para>
</sect2>

<sect2><title>Rational numbers</title>
<para>Rational numbers consist of a signed integer known as the numerator, and a positive non-zero integer known as the denominator. They are written using any of the above integer syntaxes, with the numerator first, then a <literal>/</literal>, then the denominator.</para>
</sect2>

<sect2><title>Characters</title>
<para>An IRON character can be any Unicode character. I'm going to have to look in the unicode specs for the exact terminology, but by 'character' I explicitly disallow control codes, things like BOMs, surrogates, and combining characters. Those are things used to implement characters and representational details of strings, not characters themselves.</para>
<para>A character can be represented in several ways:</para>
<itemizedlist>
  <listitem><para>A sequence of the form <literal>#'</literal> followed by the character then terminated by <literal>'</literal>. Eg, <literal>#'x'</literal></para></listitem>
  <listitem><para>A sequence of the form <literal>#ucs(</literal> followed by the UCS codepoint in hexadecimal then a closing <literal>)</literal>. Eg, <literal>#ucs(12ee)</literal>. Any combining codepoints or modifiers required to build the character can be added too, as they form one logical character, by appending them after a <literal>+</literal> sign; for example, <literal>#ucs(12ee+3c04)</literal></para></listitem>
</itemizedlist>
</sect2>

<sect2><title>Booleans</title>
<para>Booleans may have only two values, true and false. Boolean true can be written <literal>#true</literal> and false <literal>#false</literal>.</para>
</sect2>

<sect2><title>Symbols</title>
<para>An IRON symbol is a complex beast in some ways, and very simple
  in others.</para>

<para>At heart, it is a list of strings. s-expression symbols are just
  strings, but IRON symbols are contained within namespaces, which are
  hierarchical sequences of names. In fact, IRON symbols are names in
  the CARBON directory, but that's irrelevant here.</para>

<para>However, the internal structure of symbols is generally
  unimportant, except in certain low-level operations. The important
  operation upon symbols is testing them for equality, and as such,
  implementations are encouraged to "intern" symbols into a global
  hash table, and just use pointers to each symbol's unique
  representation in memory as the symbol value, so that identity
  comparison is just a pointer test.</para>

<para>Two symbols should compare equal if and only if they are
  identical lists of identical strings (case sensitive).</para>

<para>Symbols may not contain whitespace characters of any kind. No
  component of a symbol may start with a digit.</para>

<para>There are a few different structures for symbols:</para>

<itemizedlist>

  <listitem><para>The full unambigious absolute path: every element of
  the path, in order, separated by <literal>/</literal> characters,
  bracketed by <literal>&lt;</literal>
  and <literal>&gt;</literal>. Special characters
  (<literal>/</literal>,<literal>\</literal>,<literal>&lt;</literal>,
  and <literal>&gt;</literal>) in each path component can be escaped
  with prefixed <literal>\</literal> characters.</para></listitem>

  <listitem><para>Relative symbols, which refer to the current
  namespace, are just written as a sequence of one or more path
  components, separated by <literal>/</literal>,
  with <literal>:</literal>, <literal>/</literal>
  and <literal>\</literal> characters escaped with
  prefixed <literal>\</literal> characters. The actual value of the
  symbol is found by appending the supplied path components to the
  current namespace. If the first character of the symbol would
  be <literal>!</literal>, a
  digit, <literal>;</literal>, <literal>(</literal>, <literal>[</literal>, <literal>{</literal>, <literal>&lt;</literal>
  or <literal>#</literal> then it must also be escaped with a
  prefixed <literal>\</literal> character to avoid
  ambiguity.</para></listitem>

  <listitem><para>Prefixed symbols, which refer to a declared
    namespace, are written as the namespace name (which has the syntax
    of a symbol path component) followed by <literal>:</literal> then
    one or more path components, separated by <literal>/</literal>. As
    before, <literal>:</literal>, <literal>/</literal>
    and <literal>\</literal> characters may be escaped with a
    prefixed <literal>\</literal>. The actual value of the symbol is
    found by appending the supplied path components to the namespace
    bound to the supplied namespace name.</para></listitem>

</itemizedlist>

<para>When read from a written form into memory, although the symbol
  is resolved from any of the above formats into a full absolute path,
  its original representation is stored too so it can be recreated;
  the result of parsing an IRON value is the value itself plus a map
  of namespace prefixes and a default namespace.

<para>The declaration of the current namespace, or named namespaces,
  is explained later.</para>

</sect2>

<sect2><title>The empty list</title>
<para>The empty list, as in s-expressions, is written as <literal>()</literal>.</para>
</sect2>

<sect2><title>Nil</title>
<para>Nil, a value that explicitly represents the absence of a value, is written <literal>#nil</literal>.</para>
</sect2>

</sect1>

<sect1><title>Compound types</title>

  <sect2><title>Pairs</title>

  <para>IRON has pairs, also known as cons cells, just as Lisp does.</para>

  <para>Likewise, it has Lisp's syntax for pairs (and syntactic sugar for lists).</para>

  <para>A basic pair can be written as <literal>(</literal> followed by the first value, then <literal>.</literal>, then the second value, then <literal>)</literal>.</para>

  <para>Linked lists of pairs are written just as in s-expressions. <literal>(</literal> followed by a space-separated list of elements then <literal>)</literal>. Improper lists can be written as <literal>(</literal> followed by a space-separated list of elements then <literal>.</literal> followed by the tail and then <literal>)</literal>.</para>
  
  </sect2>
  
  <sect2><title>Maps</title>

    <para>We inherit more from YAML than s-expressions in having a map type in the core. Maps are notionally sets of pairs, with the constraint that no two pairs in the set may share the first element. They might be implemented as a hash table, but there are many situations in which they should not be.</para>

  <para>The written representation for them is <literal>{</literal> followed by zero or more elements, written as space-separated pairs of values, with <literal>:</literal> characters separating them for readability, and the map is terminated with <literal>}</literal>.</para>
  </sect2>

  <sect2><title>Records</title>

    <para>Records, however, inherit more from Smalltalk, at least in syntax.</para>

    <para>The notional representation of a record is as a symbol followed by a list of values. The symbol is known as the 'type' of the record, and list of zero or more values known as the 'fields'.</para>

    <para>However, the written representation is somewhat special, and attaches special meaning to colons in the last component of the type symbol.</para>

    <para>The simplest representation is for records whose type symbol has no colons in the last component. These are written as a <literal>[</literal> followed by the type symbol (in any of the symbol representations listed above), then some whitespace, then the space-separated list of fields, terminated with a <literal>]</literal>.</para>

    <para>For example, <literal>[+ 1 2 3]</literal>.</para>

    <para>However, if the last component of the type symbol contains
    colons, then the last character of the component must itself be a
    colon, or else the symbol cannot be used as a record type
    symbol. The last component of the type symbol can be considered as
    a concatenated sequence of colon-terminated strings known as the
    field names.</para>

    <para>The written representation of a record with such a type
    symbol again starts with <literal>[</literal>, but it is now
    followed by the type symbol, but only including the first field
    name, without its trailing colon. This can be written in any of
    the symbol syntaxes listed above, but it does not necessarily
    represent a symbol, merely part of a symbol. After this type
    symbol fragment we have the colon, then we have the value of the
    first field, separated by whitespace. There can then follow a
    whitespace-separated list of zero or more fields, represented as
    the field name (written as if it were a symbol relative to a
    current namespace, even though it is only a fragment of a symbol)
    followed by a colon then the field's value. The number of fields
    must be the number of field names in the record's type symbol;
    records with more or less fields than they have field names are
    invalid syntax.</para>

    <para>For example, <literal>[if: [= x 1] then: [' hello] else: ['
    goodbye]]</literal> represents a record with the type
    symbol <literal>if:then:else:</literal> (relative to the current
    namespace), and fields <literal>[= x 1]</literal>, <literal>['
    hello]</literal>, and <literal>[' goodbye]</literal>, each of
    which is itself a record.</para>

    <para>Alternatively, <literal>[&lt;foo/bar/baz&gt;: 1 bam:
    2]</literal> represents a record with the type
    symbol <literal>foo/bar/baz:bam:</literal> and
    fields <literal>1</literal> and <literal>2</literal>.</para>
    
    <para>It is perfectly legal to have a symbol that happens to end
    in a colon as a value in a record; when appearing in a value
    position it will always be parsed as a value, but it would be
    clearer on the eye to quote it with a <literal>\</literal>
    character.</para>
    
  </sect2>
  <sect2><title>Vectors</title>

    <para>Vectors are pretty similar to lists - but, don't forget, IRON has no explicit list type, just pairs. And pairs are, at best, a special case of vectors.</para>

    <para>There are two basic kinds of vectors: general and heterogenous.</para>

    <para>General vectors contain lists of arbitrary values, while heterogenous vectors place restrictions on their contents.</para>

    <para>General vectors are written as <literal>#&lt;</literal> followed by a whitespace-separated list of values, terminated by <literal>&gt;</literal>. For example, <literal>#&lt;a b 1 2 3&gt;</literal>.</para>

    <para>Heterogenous vectors are usually written
    as <literal>#</literal> followed by a type name
    then <literal>&lt;</literal>, the whitespace-separated list of
    values, then <literal>&gt;</literal>. Valid type names
    are <literal>float</literal>, <literal>symbol</literal>, <literal>u8</literal>, <literal>s8</literal>, <literal>u16</literal>, <literal>s16</literal>, <literal>u32</literal>, <literal>s32</literal>, <literal>u64</literal>, <literal>s64</literal>,
    or <literal>char</literal>. The <literal>u8</literal> and friends
    represent particular limited integer types - unsigned or signed
    (two's complement) integers of the specified number of bits. For
    example, <literal>#u8&lt;1 2 3
    4&gt;</literal>, <literal>#symbol&lt;a b c&gt;</literal>,
    or <literal>#char&lt;#&#x27;a&#x27; #&#x27;b&#x27;
    #&#x27;c&#x27;&gt;</literal>.</para>

    <para>However, vectors of characters can be represented more compactly by just enclosing the verbatim character sequence in <literal>"</literal> characters, after escaping any <literal>\</literal> or <literal>"</literal> characters in the string by prefixing them with <literal>\</literal>. Also, characters may be written as their UCS codepoints, in the form <literal>\(<replaceable>x</replaceable>)</literal> where <replaceable>x</replaceable> is the code in hex, or a series of hex codes separated with <literal>+</literal> for multi-codepoint characters. For example, the last example can be written <literal>"abc"</literal>.</para>
  </sect2>
  
  <sect2><title>Other types</title>

    <para>In general, other types can be represented by introducing extra syntax of the form <literal>#</literal> followed by a name then optionally <literal>(</literal>, some content, then <literal>)</literal>. Currently used names are <literal>true</literal>, <literal>false</literal>, <literal>nil</literal>, and <literal>ucs</literal>.</para>
  </sect2>
    
  <sect2><title>The top level</title>

    <para>Given a sequence of characters to parse, the IRON written form parser considers the sequence to be some whitespace followed by a single value. After the value has been successfully parsed, any remaining characters will be left for later consumption if the sequence is a sequential-access stream. If the sequence is a fixed-length string or other random-access character sequence, then it is an error for anything other than whitespace to remain.</para>
  </sect2>
    
  <sect2><title>Namespaces</title>

    <para>Symbols may be represented compactly in the IRON written
    form by reusing common prefixes of the symbol path, by declaring
    them as namespaces.</para>

    <para>The IRON parser maintains a current namespace
    environment, consisting of a default namespace symbol and a map
    from namespace names to namespace symbols. At the start of parsing
    an IRON written form, the default namespace is provided by the
    caller (and should be the CARBON name of the place where the IRON
    document came from, where applicable) and the map
    is <literal>{"Fe": &lt;/argon/iron&gt;}</literal>.</para>

    <para>The default namespace may be changed at the start of a
    written IRON value, before any actual value syntax appears, by
    introducing the syntax <literal>!defns</literal> followed by
    actual whitespace then a symbol which thereafter becomes the new
    default namespace. The default namespace may not be changed
    later.</para>

    <para>Named namespace bindings may be created, or existing ones
    overridden, by introducing the syntax <literal>!ns</literal>
    followed by actual whitespace, then a namespace name, some more
    actual whitespace, then a symbol which is thereafter bound to that
    name in the namespace map. The same namespace name may not be
    rebound later.</para>

</sect2>
<sect2><title>Shorthands</title>

<para>The IRON data model as no inherent notion of a comment, because these are just handled as record types that wrap the value they are commenting upon.</para>

<para>A comment is represented as a record with a type symbol of <literal>/argon/iron/note::</literal>. There are two fields, <literal>note:</literal> and <literal>:</literal>. The former is the body of the comment, and the latter is the value being commented upon.</para>

<para>As a shorthand for this unwieldy syntax, the textual representation also supports the use of the <literal>;</literal> character to mark comments. If the <literal>;</literal> is the first non-whitespace character on the line, then it is considered a comment on the next value in the stream. If it is not the first non-whitespace character on the line, then it is considered a comment on the previous value in the stream. Either way, the body of the comment is taken to be the string from the ; to the end of the line. If more than one comment applies to the same value by this rule, then they are concatenated together, separated by newlines.</para>

<para>For example:
  <programlisting>
    ; This is me
    [person: "Alaric Snell-Pym"
     speciality: "Vapourware" ; All I do is talk about it!
    ]
  </programlisting>
  
  is equivalent to:
  
  <programlisting>
    [fe:note: "This is me"
     : [person: "Alaric Snell-Pym"
        speciality: [fe:note: "All I do is talk about it!" : "Vapourware"]]]
  </programlisting>
</para>

<para>However, comments are often used for several different things; the case we have covered is actually applying notes to values, but they are often used to temporarily disable sections of something ("commenting out"). This is handled by wrapping the value in another record, with type <literal>/argon/iron/disabled::</literal>. As before, the first field is a string describing the reason for disablement, and the second field is the disabled value.</para>

<para>Often, when something is commented out, it is temporarily replaced with another value. In which case, we can make this relationship explicit by using the <literal>/argon/iron/temp:use::</literal> record. The first field is a string giving reason for the swap, the second field is the value to actually use, and the third field is the value to ignore.</para>

<para>All three of the previous record types have something in common: they are intended to wrap around an existing value. This is a common pattern with records, but it can be difficult to place the trailing <literal>]</literal> after the wrapped value if it has a large multi-line written representation and as such, we provide a shorthand representation.</para>

<para>Any record type whose last field has the minimal name <literal>:</literal> - eg, any record whose type symbol ends in <literal>::</literal> - may be represented as a prefix record. This consists of writing the record value, but omitting the final <literal>:</literal> field. However, the trailing <literal>]</literal> must be replaced by <literal>]:</literal>, which causes the next value to be read and inserted into the record as an extra field called <literal>:</literal>.</para>

<para>For example, to disable a value, one can just prefix it with <literal>[fe:disabled: "Not yet implemented"]:</literal>, and comments can be attached to objects by prefixing them with  <literal>[fe:note: "This is a silly value"]:</literal>.</para>

</sect2>

<sect2><title>Closures, Continuations, and other such tricky things</title>

<para>Closures and continuations are all eminently serialisable; both consists of a reference to the code, and a closed-over environment. However, how to refer to code is beyond the scope of this standard. As literal closures, continuations, and other hairy objects like handles to specific objects will rarely need to be typed in by hand, they do not need especially pleasant syntactic sugar, either. Therefore, such objects are left to higher levels to represent as records with appropriate type symbols.</para>

</sect2>

</sect1>

</chapter>
<chapter><title>Binary form</title>
<para>TODO: Same model as textual form, different syntax</para>
</chapter>

<chapter><title>In-memory form</title>
<para>TODO: Defined in terms of HYDROGEN. And we need to define garbage collection for it. And mechanisms for going to and from textual form, both as a SAX-like eventy interface (for reasons that will become clear later) and direct to/from objects in memory.</para>
<para>Also, linearity becomes an issue in some way. Newly loaded values are linear, but there are <function>make-shareable</function> and <function>make-linear</function> operations.</para>
</chapter>
</book>
