Magic Pipes

When Magic Pipes runs user code, the following bindings are available in the environment:

The runtime tools are described below. They can be loaded in external code by importing the magic-pipes-runtime module.

(mplog format args...)

This uses Chicken's printf formatting system to output strings to the standard error port. It's convenient to use this in Magic Pipes code to display progress, debugging, and informational reports to the user, without disrupting pipeline output to standard output.

(mplog "Current total: ~A" current-total)

(mplookup type filename [dupmode: {all|one}] [reverse: boolean])

This opens a persistent key:value lookup table. Several file types are supported, which will be described below. mplookup returns a suite of values, each of which is a procedure; in order, they are the lookup procedure, the update procedure, the deletion procedure, the fold procedure, and the close procedure. If you don't call the close procedure, not only may you leak resources, but updates and deletions you have performed may not be correctly written to the file.

The lookup procedure accepts a key, looks it up in the lookup table, and returns the corresponding value, or #f if there is none. An optional second argument can be provided, which is used as the default value instead of #f. However, if dupmode was set to all (the default is one) when the lookup table was opened, then the lookup procedure instead returns a list of matching values; this list will be empty if there are none, and can contain more than one value of the lookup table contains duplicates.

The update procedure accepts a key and value, and binds that key solely to that value in the lookup table. Any previous bindings of that key to values in the lookup table are deleted. There is currently no way to bind a key to more than one value through this interface (but I might extend it in future).

The delete procedure accepts a key, and removes any values associated to that key in the lookup table.

The fold procedure accepts a procedure of three arguments (key, value and accumulator), and an initial accumulator. It calls the procedure for every key:value binding in the lookup table (which, if dupmode is all, might be several times for a single key), threading an accumulator value through.

Finally, the close procedure writes any pending changes to the file, and releases any held resources.

If the optional reverse argument is true, then the lookup table is inverted.

Lookup table type sqlite

This lookup table type uses an SQLite database containing s-expressions, with a unique index on the key column and an index on the value column. As such, it can only represent a single value for each key. The database is created transparently if it does not already exist (the suggested extension is .sqlite). As lookup of keys and values is done by their exact textual representation, it is not recommended that the SQLite database be modified directly, as a different encoding of the same s-expression value may produce erronious results.

Lookup table type aliases

This lookup table type uses a plain text file of the sort traditionally used to specify email aliases. On each line, any hash (#) symbol and the rest of the line thereafter is ignored; from what remains, entries of the form key:value (with any whitespace before or after the key or value being ignored) are interpreted as the bindings of the lookup table, with the key and the value both being taken as strings without any parsing. Lines not matching that structure are ignored silently.

Lookup table type alist

This lookup table type uses a plain text file containing zero or more alists, written as sexprs. An alist is a list whose elements are pairs mapping keys to values, like so:

((key . value)
 (message . "Hello World")
 (complex-structure . (1 2 (3 4 5 6)))

If there are multiple alists in the same file, they are all logically concatenated. Multiple occurrences of the same key, be they in the same alist or not, are handled as per the lookup table's dupmode setting.

Lookup table type sexprs

This lookup table type is very similar to alist, except without the "outer list"; the file is read as a sequence of sexprs, each of which is a single (key . value) pair. The advantage over alist is that the resulting file is easier to process one entry at a time, without ending up reading the entire alist into memory in one go, when read or written directly rather than via mplookup.

Dirent tools

mpls reads directory entries into a structured object called a "dirent"; a number of utility procedures are provided to manipulate them.

(->dirent path-or-dirent)

If the argument is a string, creates a dirent object representing that path. An error is signalled if the path does not exist.

If the argument is already a dirent, returns it as-is.

Otherwise, an error is signalled.

(dirent? object)

Returns a true value if the supplied object is a dirent, or #f otherwise.


The dirent accessors return various attributes of the directory entry.