// Copyright Graham St Jack
module bob;

import std.stdio;
import std.ascii;
import std.string;
import std.format;
import std.process;
import std.algorithm;
import std.range;
import std.file;
import std.path;
import std.conv;
import std.datetime;
import std.getopt;
import std.concurrency;
import std.functional;
import std.exception;

import core.sys.posix.sys.wait;
import core.sys.posix.signal;


/*========================================================================

A build tool suitable for C/C++ and D code, written in D.

Objectives of this build tool are:
* Easy to write and maintain build scripts (Bobfiles):
  - Simple syntax.
  - Automatic determination of which in-project libraries to link.
* Auto execution and evaluation of unit tests.
* Enforcement of dependency rules.
* Support for building source code from multiple repositories.
* Support for C/C++ and D.
* Support for code generation:
  - A source file isn't scanned for imports/includes until after it is up to date.
  - Dependencies inferred from these imports are automatically applied.


Directory Structure
-------------------

The source code is arranged into repositories, and within repositories
into packages which correspond to the directory structure.

Packages should contain not only library source code, but also tests, utilities,
documentation and data. The intention is to make packages easily reusable
by putting all their pieces in one location.

A typical directory structure is:

repo                         repository
  |
  +--package-name      package's Bobfile and library source
        |
        +--doc         the package's documentation
        +--data        data files needed by the package
        +--test        source code for automated regression tests
        +--util        source code for non-test executables
        |
        +--child-package(s)


Dependency rules
----------------

Files and their owning packages are arranged in a tree with cross-linking
dependencies. Each node in the tree can be public or protected. The root of the
tree contains its children publicly.

The dependency rules are:
* A protected node can only be referred to by sibling nodes or nodes contained
  by those siblings.
* A node can only refer to another node if its parent transitively refers to or
  transitively contains that other node.
* Circular dependencies are not allowed.

An object file can only be used once - either in a library or an executable.
Dynamic libraries don't count as a use - they are just a repackaging.

A dynamic library cannot contain the same static library as another dynamic library.


Configure
---------

Bundled with bob is a configure procedure that is mainly implemented in
configure_functions.d. Each project contains a configure.d program that
is invoked via a configure.sh script.

Configure establishes a build directory complete with scripts necessary
to build and run a project.

Refer to configure_functions.d for details.


Bobfiles
--------

Each package has a Bobfile. On start, bob (the builder) reads the
Bobfile specified in the Boboptions file and follows its directives.

Bobfiles must refer to things in dependency order, so that anything referred
to is defined earlier.


General rule format in Bobfiles is:

# This line is a comment.

# General form of a rule. Arguments are space-separated tokens. Args are optional.
rulename target : sources : arg1 : arg2 : arg3;

# Resolves to 0 or 1 rule, choosing the first that matches an architecture specified
# during configure. [] is a default that matches regardless of architecture.
ARCHITECTURE{
    [arch-1] rulename target : arg1 : arg2 : arg3;
    [arch-2] rulename target : arg1 : arg2 : arg3;
    []       rulename target : arg1 : arg2 : arg3;
}


Specific rules are:

# This package contains packages. eg: package math ipc;
package names [: protected];  # defaults to public

# This package refers to others. eg: refer math ipc;
# Referenced packages are specified by path relative to root. eg: icp/client
refer packages;

# Library: for D/C/C++, list both header and body files, all of which must be local.
# eg: static-lib math : matrix.h : matrix.cc : m;
static-lib  name : public-sources : protected-sources : required-system-libs;

# dynamic-libs can only contain static-libs within their package or its
# children. The contained static-libs are specified as a relative name trail,
# with the last element optionally omitted if it is the same as the package name.
# eg: dynamic-lib ipc : common client server;
dynamic-lib name : static-libs;
dynamic-lib name : static-libs : plugin;


# Executables: makes an exe from a single source file and any libs deduced from
# imports and includes. Also links with specified system libs, plus any specified
# by deduced libraries.
# Tests are auto-executed and build fails on test error.
test-util name : source : required-system-libs; # source file in test dir or generated
priv-util name : source : required-system-libs; # source file in util dir or generated
dist-util name : source : required-system-libs; # source file in util dir or generated

# Shell
priv-shell name; # source file in util dir or generated
dist-shell name; # source file in util dir or generated

# Data - any dirs named specify whole contained tree, not including symlinks
test-data name;  # source in data dir or generated.
dist-data name;  # source in data dir or generated.

# Docco - rst files converted to html
doc names;        # source in doc dir or generated.
doc-data names;   # source in doc dir or generated.


Build directory structure
-------------------------

Bob assumes that a build directory has been established
by an earlier configure operation. The configure sets up
a Boboptions file containing definitions of variables used by rules.

The format of Boboptions is:
key = value;

Usually bob is invoked from a build script established during configure.

Under the build directory we have:

obj
  static-libs
  packages(s)
    obj-files
    generated-sources
    packages(s)
priv
  package(s)
    test-exes
    util-exes
    test
      test-exe-specific-test-dir(s)
        test-results
        test-working-files
    doc
      docs
    package(s)
dist
  data      All dist-data
  lib       All dynamic-libs except plugins
    plugin  Dynalic-libs designated as plugins
  bin       All dist-util



Search paths
------------

Compilers are told to look in 'src' and 'obj' directories for input files.
The src directory contains links to each top-level package in all the
repositories that comprise the project.

Therefore, include directives have to include the path starting from
the top-level package names, which must be unique.

This namespacing avoids problems of duplicate filenames
at the cost of the compiler being able to find everything, even files
that should not be accessible. Bob therefore enforces all visibility
rules before invoking the compiler.


The build process
-----------------

Bob reads the project Bobfile, transiting into
other-package Bobfiles as packages are mentioned.

Bob assumes that new packages, libraries, etc
are mentioned in dependency order. That is, when each thing is
mentioned, everything it depends on, including dependencies inferred by
include/import statements in source code, has already been mentioned.

The planner scans the Bobfiles, binding files to specific
locations in the filesystem as it goes, and builds the dependency graph.

The file state sequence is:
    initial
    dependencies_clean         skipped if no dependencies
    building                   skipped if no build action
    up-to-date
    scanning_for_includes
    includes_known
    clean

As files become buildable, actions are passed to workers.

Results cause the dependency graph to be updated, allowing more actions to
be issued. Specifically, generated source files are scanned for import/include
after they are up to date, and the dependency graph and action commands are
adjusted accordingly.

//==========================================================================*/

/+
// signal handler for otherwise-fatal thread-specific signals 
extern (C) void threadSpecificSignalHandler(int signum) {
    string name() {
        switch (signum) {
        case SIGSEGV: return "SIGSEGV";
        case SIGFPE:  return "SIGFPE";
        case SIGILL:  return "SIGILL";
        case SIGABRT: return "SIGABRT";
        default:      return "";
        }
    }

    writefln("called!!!");
    stdout.flush;
    writefln("got signal %s %s", signum, name);
    stdout.flush;
    throw new Error(format("Got signal %s %s", signum, name));
}

// install a signal handler
sigaction_t setHandler(int signum, sigfn_t handler) {
    sigset_t empty_mask;
    sigemptyset(&empty_mask);

    sigaction_t new_action;
    new_action.sa_handler = handler;
    new_action.sa_mask    = empty_mask;
    new_action.sa_flags   = SA_RESETHAND;

    sigaction_t old_action;

    sigaction(signum, &new_action, &old_action);

    return old_action;
}

shared static this() {
    writefln("setting up thread-specific signal handlers");

    // set up shared signal handlers for fatal thread-specific signals
    setHandler(SIGFPE,  &threadSpecificSignalHandler);
    setHandler(SIGILL,  &threadSpecificSignalHandler);
    setHandler(SIGSEGV, &threadSpecificSignalHandler);

    // unblock the signals
    sigset_t unblock_set, prev_set;
    sigemptyset(&unblock_set);
    sigaddset(&unblock_set, SIGILL);
    sigaddset(&unblock_set, SIGSEGV);
    sigaddset(&unblock_set, SIGFPE);
    pthread_sigmask(SIG_UNBLOCK, &unblock_set, &prev_set);
}
+/


//-----------------------------------------------------------------------------------------
// PriorityQueue - insert items in any order, and remove largest-first
// (or smallest-first if "a > b" is passed for less).
//
// A simplified adaptation of std.container.BinaryHeap. The original doesn't
// have the behaviour needed here.
//
// It is a simple input range (empty(), front() and popFront()).
//
// Notes from Wikipedia article on Binary Heap:
// * Tree is concocted using index arithetic on underlying array, as follows:
//   First layer is 0. Second is 1,2. Third is 3,4,5,6, etc.
//   Therefore parent of index i is (i-1)/2 and children of index i are 2*i+1 and 2*i+2
// * Tree is balanced, with incomplete population to right of bottom layer.
// * A parent is !less all its children.
// * Insert:
//   - Append to array.
//   - Swap new element with parent until parent !less child.
// * Remove:
//   - Replace root with the last element and reduce the length of the array.
//   - If moved element is less than a child, swap with largest child.
//-----------------------------------------------------------------------------------------
struct PriorityQueue(T, alias less = "a < b") {
private:

    T[]    _store;  // underlying store, whose length is the queue's capacity
    size_t _used;   // the used length of _store

    alias binaryFun!(less) comp;

public:

    @property size_t        length()   const nothrow { return _used; }
    @property size_t        capacity() const nothrow { return _store.length; }
    @property bool          empty()    const nothrow { return !length; }

    @property const(T)      front()    const         { enforce(!empty); return _store[0]; }

    // Insert a value into the queue
    size_t insert(T value)
    {
        // put the new element at the back of the store
        if ( length == capacity) {
            _store.length = (capacity + 1) * 2;
        }
        _store[_used] = value;

        // percolate-up the new element
        for (size_t n = _used; n; )
        {
            auto parent = (n - 1) / 2;
            if (!comp(_store[parent], _store[n])) break;
            swap(_store[parent], _store[n]);
            n = parent;
        }
        ++_used;
        return 1;
    }

    void popFront()
    {
        enforce(!empty);

        // replace the front element with the back one
        if (_used > 1) {
            _store[0] = _store[_used-1];
        }
        --_used;

        // percolate-down the front element (which used to be at the back)
        size_t parent = 0;
        for (;;)
        {
            auto left = parent * 2 + 1, right = left + 1;
            if (right > _used) {
                // no children - done
                break;
            }
            if (right == _used) {
                // no right child - possibly swap parent with left, then done
                if (comp(_store[parent], _store[left])) swap(_store[parent], _store[left]);
                break;
            }
            // both left and right children - swap parent with largest of itself and left or right
            auto largest = comp(_store[parent], _store[left])
                ? (comp(_store[left], _store[right])   ? right : left)
                : (comp(_store[parent], _store[right]) ? right : parent);
            if (largest == parent) break;
            swap(_store[parent], _store[largest]);
            parent = largest;
        }
    }
}


//------------------------------------------------------------------------------
// Synchronized object that launches external processes in the background
// and keeps track of their PIDs. A one-shot bail() method kills all those
// launched processes and prevents any more from being launched.
//
// bail() is called by the error() functions, which then throw an exception.
//
// We also install a signal handler to bail on receipt of various signals.
//------------------------------------------------------------------------------

class BailException : Exception {
    this() {
        super("Bail");
    }
}

synchronized class Launcher {
    private {
        bool      bailed;
        bool[int] children;
    }

    // launch a process if we haven't bailed
    pid_t launch(string command) {
        if (bailed) {
            say("Aborting launch because we have bailed");
            throw new BailException();
        }
        int child = spawnvp(P_NOWAIT, "/bin/bash", ["bash", "-c", command]);
        //say("spawned child, pid=%s", child);
        children[child] = true;
        return child;
    }

    // a child has been finished with
    void completed(pid_t child) {
        children.remove(child);
        //say("completed child, pid=%s", child);
    }

    // bail, doing nothing if we had already bailed
    bool bail() {
        if (!bailed) {
            bailed = true;
            foreach (child; children.byKey) {
                //say("killing child, pid=%s", child);
                kill(child, SIGTERM);
            }
            return false;
        }
        else {
            return true;
        }
    }
}

shared Launcher launcher;

void doBail() {
    launcher.bail;
}
extern (C) void mySignalHandler(int sig) {
    // launch a thread to initiate a bail
    say("got signal %s", sig);
    spawn(&doBail);
}


shared static this() {
    // set up shared Launcher and signal handling

    launcher = new shared(Launcher)();

    /*
    signal(SIGTERM, &mySignalHandler);
    signal(SIGINT,  &mySignalHandler);
    signal(SIGHUP,  &mySignalHandler);
    */
}



//------------------------------------------------------------------------------
// printing utility functions
//------------------------------------------------------------------------------

// where something originated from
struct Origin {
    string path;
    uint   line;
}

private void sayNoNewline(A...)(string fmt, A a) {
    auto w = appender!(char[])();
    formattedWrite(w, fmt, a);
    stderr.write(w.data);
}

void say(A...)(string fmt, A a) {
    auto w = appender!(char[])();
    formattedWrite(w, fmt, a);
    stderr.writeln(w.data);
    stderr.flush;
}

void fatal(A...)(string fmt, A a) {
    say(fmt, a);
    launcher.bail;
    throw new BailException();
}

void error(A...)(ref Origin origin, string fmt, A a) {
    sayNoNewline("%s|%s| ERROR: ", origin.path, origin.line);
    fatal(fmt, a);
}

void errorUnless(A ...)(bool condition, Origin origin, lazy string fmt, lazy A a) {
    if (!condition) {
        error(origin, fmt, a);
    }
}


//-------------------------------------------------------------------------
// path/filesystem utility functions
//-------------------------------------------------------------------------

//
// Return the given path with the top level directory removed
//
string clip(string path) {
    for (uint i = 0; i < path.length; ++i) {
        if (isDirSeparator(path[i])) return path[i+1..$];
    }
    return "";
}


//
// Ensure that the parent dir of path exists
//
void ensureParent(string path) {
    static bool[string] doesExist;

    string dir = dirName(path);
    if (dir !in doesExist) {
        if (!exists(dir)) {
            ensureParent(dir);
            say("%-15s %s", "Mkdir", dir);
            mkdir(dir);
        }
        else if (!isDir(dir)) {
            error(Origin(), "%s is not a directory!", dir);
        }
        doesExist[path] = true;
    }
}


//
// return the modification time of the file at path
// Note: A zero-length target file is treated as if it doesn't exist.
//
long modifiedTime(string path, bool isTarget) {
    if (!exists(path) || (isTarget && getSize(path) == 0)) {
        return 0;
    }
    SysTime fileAccessTime, fileModificationTime;
    getTimes(path, fileAccessTime, fileModificationTime);
    return fileModificationTime.stdTime;
}


// return the privacy implied by args
Privacy privacyOf(ref Origin origin, string[] args) {
    if (!args.length ) return Privacy.PUBLIC;
    else if (args[0] == "protected") return Privacy.PROTECTED;
    else if (args[0] == "semi-protected") return Privacy.SEMI_PROTECTED;
    else if (args[0] == "private")   return Privacy.PRIVATE;
    else if (args[0] == "public")    return Privacy.PUBLIC;
    else error(origin, "privacy must be one of public, semi-protected, protected or private");
    assert(0);
}




//------------------------------------------------------------------------
// File parsing
//------------------------------------------------------------------------

// options read from Boboptions file
string[string] options;
bool[string]   architectures;
bool[string]   validArchitectures;

//
// Read an options file, populating options
// Format is:   key=value;\n
// value can contain '='
//
void readOptions() {
    string path = "Boboptions";
    Origin origin = Origin(path, 1);

    errorUnless(exists(path) && isFile(path), origin, "can't read Boboptions %s", path);

    string content = readText(path);

    string key;
    int anchor = 0;
    foreach (int pos, char ch ; content) {
        if (ch == '\n') ++origin.line;
        if (key is null && ch == '=') {
            key = strip(content[anchor..pos]);
            anchor = pos + 1;
        }
        else if (ch == ';') {
            if (key is null) error(origin, "option terminated without a key");
            string str = strip(content[anchor..pos]);
            if (key == "VALID_ARCHITECTURES") {
                foreach (arch; split(str)) {
                    validArchitectures[arch] = true;
                }
            }
            else if (key == "ARCHITECTURES") {
                foreach (arch; split(str)) {
                    architectures[arch] = true;
                }
            }
            options[key] = str;
            key = null;
        }
        else if (ch == '\n') {
            errorUnless(key is null, origin, "option %s not terminated with ';'", key);
            anchor = pos + 1;
        }
    }
    errorUnless(key is null, origin, "%s ends in unterminated option", path);
}

string getOption(string key) {
    auto value = key in options;
    if (value) {
        return *value;
    }
    else {
        return "";
    }
}



//
// Scan file for includes, returning an array of included trails
//   #   include   "trail"
//
// All of the files found should have trails relative to "src" (if source)
// or "obj" (if generated). All system includes must use angle-brackets,
// and are not returned from a scan.
//
struct Include {
    string trail;
    uint   line;
}

Include[] scanForIncludes(string path) {
    Include[] result;
    Origin origin = Origin(path, 1);

    enum Phase { START, HASH, WORD, INCLUDE, QUOTE, NEXT }

    if (exists(path) && isFile(path)) {
        string content = readText(path);
        int anchor = 0;
        Phase phase = Phase.START;

        foreach (int i, char ch; content) {
            if (ch == '\n') {
                phase = Phase.START;
                ++origin.line;
            }
            else {
                switch (phase) {
                case Phase.START:
                    if (ch == '#') {
                        phase = Phase.HASH;
                    }
                    else if (!isWhite(ch)) {
                        phase = Phase.NEXT;
                    }
                    break;
                case Phase.HASH:
                    if (!isWhite(ch)) {
                        phase = Phase.WORD;
                        anchor = i;
                    }
                    break;
                case Phase.WORD:
                    if (isWhite(ch)) {
                        if (content[anchor..i] == "include") {
                            phase = Phase.INCLUDE;
                        }
                        else {
                            phase = Phase.NEXT;
                        }
                    }
                    break;
                case Phase.INCLUDE:
                    if (ch == '"') {
                        phase = Phase.QUOTE;
                        anchor = i+1;
                    }
                    else if (!isWhite(ch)) {
                        phase = Phase.NEXT;
                    }
                    break;
                case Phase.QUOTE:
                    if (ch == '"') {
                        result ~= Include(content[anchor..i].idup, origin.line);
                        phase = Phase.NEXT;
                    }
                    break;
                case Phase.NEXT:
                    break;
                default:
                    error(origin, "invalid phase");
                }
            }
        }
    }
    return result;
}


//
// Scan a D source file for imports.
//
// The parser is simple and fast, but can't deal with version
// statements or mixins. This is ok for now because it only needs
// to work for source we have control over.
//
// The approach is:
// * Scan for a line starting with "static", "public", "private" or ""
//   followed by "import".
// * Then look for:
//     ':' - module is previous word, and then skip to next ';'.
//     ',' - module is previous word.
//     ';' - module is previous word.
//   The import is terminated by a ';'.
//

Include[] scanForImports(string path) {
    Include[] result;
    string content = readText(path);
    string word;
    int anchor, line=1;
    bool inWord, inImport, ignoring;

    string[] externals = split(getOption("DEXTERNALS"));

    foreach (int pos, char ch; content) {
        if (ch == '\n') {
            line++;
        }
        if (ignoring) {
            if (ch == ';' || ch == '\n') {
                // resume looking for imports
                ignoring = false;
                inWord   = false;
                inImport = false;
            }
            else {
                // ignore
            }
        }
        else {
            // we are not ignoring

            if (inWord && (isWhite(ch) || ch == ':' || ch == ',' || ch == ';')) {
                inWord = false;
                word = content[anchor..pos];

                if (!inImport) {
                    if (isWhite(ch)) {
                        if (word == "import") {
                            inImport = true;
                        }
                        else if (word != "public" && word != "private" && word != "static") {
                            ignoring = true;
                        }
                    }
                    else {
                        ignoring = true;
                    }
                }
            }

            if (inImport && word && (ch == ':' || ch == ',' || ch == ';')) {
                // previous word is a module name

                string trail = std.array.replace(word, ".", dirSeparator) ~ ".d";

                bool ignored = false;
                foreach (external; externals) {
                    string ignoreStr = external ~ dirSeparator;
                    if (trail.length >= ignoreStr.length && trail[0..ignoreStr.length] == ignoreStr) {
                        ignored = true;
                        break;
                    }
                }

                if (!ignored) {
                    result ~= Include(trail, line);
                }
                word = null;

                if      (ch == ':') ignoring = true;
                else if (ch == ';') inImport = false;
            }

            if (!inWord && !(isWhite(ch) || ch == ':' || ch == ',' || ch == ';')) {
                inWord = true;
                anchor = pos;
            }
        }
    }

    return result;
}


//
// read a Bobfile, returning all its statements
//
// //  a simple statement
// rulename targets... : arg1... : arg2... : arg3...; // can expand Boboptions variable with ${var-name}
//
// // a conditional statement resolving to 0 or one of its constituents, based on ARCHITECTURE Boboption.
// ARCHITECTURE{
//   [arch-1] rulename targets... : arg1... : arg2... ;
//   [arch-2] rulename targets... : arg1... : arg2... ;
//   []       rulename targets... : arg1... : arg2... ; // optional default
// }
//
//

struct Statement {
    Origin   origin;
    int      phase;    // 0==>empty, 1==>rule populated, 2==rule,targets populated, etc
    string   rule;
    string[] targets;
    string[] arg1;
    string[] arg2;
    string[] arg3;

    string toString() const {
        string result;
        if (phase >= 1) result ~= rule;
        if (phase >= 2) result ~= format(" : %s", targets);
        if (phase >= 3) result ~= format(" : %s", arg1);
        if (phase >= 4) result ~= format(" : %s", arg2);
        if (phase >= 5) result ~= format(" : %s", arg3);
        return result;
    }
}

Statement[] readBobfile(string path) {
    Statement[] statements;
    Origin origin = Origin(path, 1);
    errorUnless(exists(path) && isFile(path), origin, "can't read Bobfile %s", path);

    string content = readText(path);
    Statement statement;

    int anchor = 0;
    bool inWord = false;
    bool inComment = false;
    bool inCondition = false;
    bool conditionMatch = false;
    bool conditionSatisfied = false;

    foreach (int pos, char ch ; content) {
        if (ch == '\n') {
            ++origin.line;
        }
        if (ch == '#') {
            inComment = true;
            inWord = false;
        }
        if (inComment) {
            if (ch == '\n') {
                inComment = false;
                anchor = pos;
            }
        }
        else if ((isWhite(ch) || ch == ':' || ch == ';')) {
            if (inWord) {
                inWord = false;
                string word = content[anchor..pos];

                if (word == "ARCHITECTURE{") {
                    // start an architecture condition
                    errorUnless(!inCondition, origin,
                                 "nested ARCHITECTURE condition in %s", path);
                    errorUnless(statement.phase == 0, origin,
                                 "ARCHITECTURE condition inside a statement in %s", path);
                    inCondition = true;
                    conditionMatch = false;
                    conditionSatisfied = false;
                }
                else if (word.length >= 2 && word[0] == '[' && word[$-1] == ']') {
                    // architecture specifier preceeds statement
                    string architecture = word[1..$-1];
                    errorUnless(inCondition, origin,
                                 "architecture '%s' specifier when not in condition in %s",
                                 architecture, path);
                    errorUnless(statement.phase == 0, origin,
                                 "nested statements in %s near %s", path, word);
                    errorUnless(architecture == "" || architecture in validArchitectures, origin,
                                 "invalid architecture '%s' in %s", architecture, path);
                    if (!conditionSatisfied && (architecture == "" || architecture in architectures)) {
                        conditionMatch = true;
                    }
                }
                else if (word == "}") {
                    inCondition = false;
                    conditionMatch = false;
                    conditionSatisfied = false;
                }
                else {
                    // should be a word in a statement

                    string[] words = [word];

                    if (word.length > 3 && word[0..2] == "${" && word[$-1] == '}') {
                        // macro substitution
                        words = split(getOption(word[2..$-1]));
                    }

                    if (word.length > 0) {
                        if (statement.phase == 0) {
                            statement.origin = origin;
                            statement.rule = words[0];
                            ++statement.phase;
                        }
                        else if (statement.phase == 1) {
                            statement.targets ~= words;
                        }
                        else if (statement.phase == 2) {
                            statement.arg1 ~= words;
                        }
                        else if (statement.phase == 3) {
                            statement.arg2 ~= words;
                        }
                        else if (statement.phase == 4) {
                            statement.arg3 ~= words;
                        }
                        else {
                            error(origin, "Too many arguments in %s", path);
                        }
                    }
                }
            }

            if (ch == ':' || ch == ';') {
                ++statement.phase;
                if (ch == ';') {
                    if (statement.phase > 1 && (!inCondition || conditionMatch)) {
                        statements ~= statement;
                        if (inCondition) {
                            conditionSatisfied = true;
                            conditionMatch = false;
                            //say("conditional resolves to statement '%s'", statement.toString);
                        }
                    }
                    statement = statement.init;
                }
            }
        }
        else if (!inWord) {
            inWord = true;
            anchor = pos;
        }
    }
    errorUnless(statement.phase == 0, origin, "%s ends in unterminated rule", path);
    return statements;
}


//-------------------------------------------------------------------------
// Planner
//
// Planner reads Bobfiles, understands what they mean, builds
// a tree of packages, etc, understands what it all means, enforces rules,
// binds everything to filenames, discovers modification times, scans for
// includes, and schedules actions for processing by the worker.
//
// Also receives results of successful actions from the Worker,
// does additional scanning for includes, updates modification
// times and schedules more work.
//
// A critical feature is that scans for includes are deferred until
// a file is up-to-date.
//-------------------------------------------------------------------------


// some thread-local "globals" to make things easier
bool g_print_rules;
bool g_print_deps;
bool g_print_details;


//
// Action - specifies how to build some files, and what they depend on
//
final class Action {
    static Action[string]       byName;
    static int                  nextNumber;
    static PriorityQueue!Action queue;

    string  name;    // the name of the action
    string  command; // the action command-string
    int     number;  // influences build order
    File[]  builds;  // files that this action builds
    File[]  depends; // files that the action's targets depend on
    bool    issued;  // true if the action has been issued to a worker

    this(ref Origin origin, string name_, string command_, File[] builds_, File[] depends_) {
        name     = name_;
        command  = command_;
        number   = nextNumber++;
        builds   = builds_;
        depends  = depends_;
        errorUnless(!(name in byName), origin, "Duplicate command name=%s", name);
        byName[name] = this;

        // add the bobfile responsible for all the files built by this action
        File bobfile;
        foreach (build; builds) {
            File b = build.bobfile;
            assert(b !is null);
            if (bobfile is null) bobfile = b;
            assert(b is bobfile);
        }
        assert(bobfile !is null);
        depends ~= bobfile;

        // set up reverse dependencies between builds and depends
        foreach (depend; depends) {
            foreach (built; builds) {
                depend.dependedBy[built] = true;
                if (g_print_deps) say("%s depends on %s", built.path, depend.path);
            }
        }
    }

    // add an extra depend to this action
    void addDependency(File depend) {
        if (issued) fatal("Cannot add a dependancy to issued action %s", this);
        if (builds.length != 1) {
            fatal("cannot add a dependency to an action that builds more than one file: %s", name);
        }
        depends ~= depend;

        // set up references and reverse dependencies between builds and depend
        foreach (built; builds) {
            depend.dependedBy[built] = true;
            if (g_print_deps) say("%s depends on %s", built.path, depend.path);
        }
    }

    // augment this action's command string with some text
    void augment(string augmentation) {
        assert(!issued);
        command ~= augmentation;
    }


    // issue this action
    void issue() {
        assert(!issued);
        issued = true;
        queue.insert(this);
    }

    override string toString() {
        return name;
    }
    override int opCmp(Object o) const {
        // reverse order
        if (this is o) return 0;
        Action a = cast(Action)o;
        if (a is null) return  -1;
        return a.number - number;
    }
}


//
// SysLib - represents a library outside the project.
//
final class SysLib {
    static SysLib[string] byName;

    string name;

    // assorted Object overrides for printing and use as an associative-array key
    override string toString() const {
        return name;
    }

    this(string name_) {
        name = name_;
        byName[name] = this;
    }
}


//
// Node - abstract base class for things in an ownership tree
// with cross-linked dependencies. Used to manage allowed references.
//

// additional constraint on allowed references
enum Privacy { PUBLIC,           // no additional constraint
               SEMI_PROTECTED,   // only accessable to descendents of grandparent
               PROTECTED,        // only accessible to children of parent
               PRIVATE }         // not accessible

class Node {
    static Node[string] byTrail;

    string  name;  // simple name this node adds to parent
    string  trail; // slash-separated name components from root to this
    Node    parent;
    Privacy privacy;
    Node[]  children;
    Node[]  refers;

    // assorted Object overrides for printing and use as an associative-array key
    override string toString() const {
        return trail;
    }

    // create the root of the tree
    this() {
        trail = "root";
        assert(trail !in byTrail, "already have root node");
        byTrail[trail] = this;
    }

    // create a node and place it into the tree
    this(ref Origin origin, Node parent_, string name_, Privacy privacy_) {
        assert(parent_);
        errorUnless(dirName(name_) == ".", origin, "Cannot define node with multi-part name '%s'", name_);
        parent  = parent_;
        name    = name_;
        privacy = privacy_;
        if (parent.parent) {
            // child of non-root
            trail = buildPath(parent.trail, name);
        }
        else {
            // child of the root
            trail = name;
        }
        parent.children ~= this;
        errorUnless(trail !in byTrail, origin, "%s already known", trail);
        byTrail[trail] = this;
    }

    // return true if this is a descendant of other
    private bool isDescendantOf(Node other) {
        for (auto node = this; node !is null; node = node.parent) {
            if (node is other) return true;
        }
        return false;
    }

    // return true if this is a visible descendant of other
    private bool isVisibleDescendantOf(Node other, Privacy allowed) {
        for (auto node = this; node !is null; node = node.parent) {
            if (node is other)            return true;
            if (node.privacy > allowed)   break;
            if (allowed > Privacy.PUBLIC) allowed--;
        }
        return false;
    }

    // return true if other is a visible-child or reference of this,
    // or is a visible-descendant of them
    private bool allowsRefTo(ref Origin origin,
                             Node       other,
                             size_t     depth        = 0,
                             Privacy    allowPrivacy = Privacy.PROTECTED,
                             bool[Node] checked      = null) {
        errorUnless(depth < 100, origin, "circular reference involving %s referring to %s", this, other);
        //say("for %s: checking if %s allowsReferenceTo %s", origin.path, this, other);
        if (other is this || other.isVisibleDescendantOf(this, allowPrivacy)) {
            if (g_print_details) say("%s allows reference to %s via containment", this, other);
            return true;
        }
        foreach (node; refers) {
            // referred-to nodes grant access to their public children, and referred-to
            // suiblings grant access to their semi-protected children
            if (node !in checked) {
                checked[node] = true;
                if (node.allowsRefTo(origin,
                                     other,
                                     depth+1,
                                     node.parent is this.parent ? Privacy.SEMI_PROTECTED : Privacy.PUBLIC,
                                     checked)) {
                    if (g_print_details) say("%s allows reference to %s via explicit reference", this, other);
                    return true;
                }
            }
        }
        return false;
    }

    // Add a reference to another node. Cannot refer to:
    // * Nodes that aren't defined yet.
    // * Self.
    // * Ancestors.
    // * Nodes whose selves or ancestors have not been referred to by our parent.
    // Also can't explicitly refer to children - you get that implicitly.
    final void addReference(ref Origin origin, Node other, string cause = null) {
        errorUnless(other !is null,                         origin, "%s cannot refer to NULL node", this);
        errorUnless(other != this,                          origin, "%s cannot refer to self", this);
        errorUnless(!this.isDescendantOf(other),            origin, "%s cannot refer to ancestor %s", this, other);
        errorUnless(!other.isDescendantOf(this),            origin, "%s cannnot explicitly refer to descendant %s", this, other);
        errorUnless(this.parent.allowsRefTo(origin, other), origin, "Parent %s does not allow %s to refer to %s", parent, this, other);
        errorUnless(!other.allowsRefTo(origin, this),       origin, "%s cannot refer to %s because of a circularity", this, other);
        if (g_print_deps) say("%s refers to %s%s", this, other, cause);
        refers ~= other;
    }
}


//
// Pkg - a package. Has a Bobfile, assorted source and built files, and sub-packages
// Used to group files together for dependency control, and to house a Bobfile.
//
final class Pkg : Node {

    File bobfile;

    this(ref Origin origin, Node parent_, string name_, Privacy privacy_) {
        super(origin, parent_, name_, privacy_);
        bobfile = File.addSource(origin, this, "Bobfile", Privacy.PRIVATE, false);
    }
}



//
// A file
//
class File : Node {
    static File[string]   byPath;       // Files by their path
    static bool[File]     allActivated; // all activated files
    static bool[File]     outstanding;  // outstanding buildable files
    static int            nextNumber;

    // Statistics
    static uint numBuilt;              // number of files targeted
    static uint numUpdated;            // number of files successfully updated by actions

    string     path;                   // the file's path
    int        number;                 // order of file creation
    bool       scannable;              // true if the file and its includes should be scanned for includes
    bool       built;                  // true if this file will be built by an action
    Action     action;                 // the action used to build this file (null if non-built)

    long       modTime;                // the modification time of the file
    bool[File] dependedBy;             // Files that depend on this
    bool       used;                   // true if this file has been used already

    // state-machine stuff
    bool       activated;              // considered by touch() for files with an action
    bool       scanned;                // true if this has already been scanned for includes
    File[]     includes;               // the Files this includes
    bool[File] includedBy;             // Files that include this
    bool       clean;                  // true if usable by higher-level files
    long       includeModTime;         // transitive max of mod_time and includes include_mod_time

    // analysis stuff
    File       youngestDepend;
    File       youngestInclude;


    // return a prospective path to a potential file.
    static string prospectivePath(string start, Node parent, string extra) {
        Node node = parent;
        while (node !is null) {
            Pkg pkg = cast(Pkg) node;
            if (pkg) {
                return buildPath(start, pkg.trail, extra);
            }
            node = node.parent;
        }
        fatal("prospective file %s's parent %s has no package in its ancestry", extra, parent);
        assert(0);
    }

    // return the bobfile that this file is declared in
    final File bobfile() {
        Node node = parent;
        while (node) {
            Pkg pkg = cast(Pkg) node;
            if (pkg) {
                return pkg.bobfile;
            }
            node = node.parent;
        }
        fatal("file %s has no package in its ancestry", this);
        assert(0);
    }

    this(ref Origin origin, Node parent_, string name_, Privacy privacy_, string path_, bool scannable_, bool built_) {
        super(origin, parent_, name_, privacy_);

        path      = path_;
        scannable = scannable_;
        built     = built_;

        number    = nextNumber++;

        modTime   = modifiedTime(path, built);

        errorUnless(path !in byPath, origin, "%s already defined", path);
        byPath[path] = this;

        if (built) {
            //say("built file %s", path);
            ++numBuilt;
        }

    }

    // Add a source file specifying its trail within its package
    static File addSource(ref Origin origin, Node parent, string extra, Privacy privacy, bool scannable) {

        // three possible paths to the file
        string path1 = prospectivePath("obj", parent, extra);  // a built file in obj directory tree
        string path2 = prospectivePath("src", parent, extra);  // a source file in src directory tree
        string path3 = baseName(extra);                        // a configure-generated source file in build directory

        string name  = baseName(extra);

        File * file = path1 in byPath;
        if (file) {
            // this is a built source file we already know about
            errorUnless(!file.used, origin, "%s has already been used", path1);
            return *file;
        }
        else if (exists(path2)) {
            // a source file under src
            return new File(origin, parent, name, privacy, path2, scannable, false);
        }
        else if (exists(path3)) {
            // a source file in build dir
            return new File(origin, parent, name, privacy, path3, scannable, false);
        }
        else {
            error(origin, "Could not find source file %s in %s, %s or %s", name, path1, path2, path3);
            assert(0);
        }
    }

    // This file has been updated
    final void updated() {
        ++numUpdated;
        modTime = modifiedTime(path, true);
        if (g_print_details) say("Updated %s, mod_time %s", this, modTime);
        if (action !is null) {
            action = null;
            outstanding.remove(this);
        }
        touch;
    }

    // Scan this file for includes, returning them after making sure those files
    // themselves exist and have already been scanned for includes
    private void scan() {
        errorUnless(!scanned, Origin(path, 1), "%s has been scanned for includes twice!", this);
        scanned = true;
        if (scannable) {

            // scan for includes that are part of the project, and thus must already be known
            Include[] entries;
            string ext = extension(path);
            if (ext == ".c" || ext == ".cc" || ext == ".h") {
                entries = scanForIncludes(path);
            }
            else if (ext == ".d") {
                entries = scanForImports(path);
            }
            else {
                fatal("Don't know how to scan %s for includes/imports", path);
            }

            foreach (entry; entries) {
                // verify that we know the included file

                File * file;
                // under src using full trail?
                File * include = cast(File *) (buildPath("src", entry.trail) in byPath);
                if (include is null && baseName(entry.trail) == entry.trail) {
                    // in src dir of the including file's package?
                    include = cast(File *) (prospectivePath("src", parent, entry.trail) in byPath);
                }
                if (include is null) {
                    // under obj?
                    include = cast(File *) (buildPath("obj", entry.trail) in byPath);
                }
                if (include is null) {
                    // build dir?
                    include = entry.trail in byPath;
                }
                Origin origin = Origin(this.path, entry.line);
                errorUnless(include !is null, origin, "included/imported unknown file %s", entry.trail);

                // add the included file to this file's includes
                includes ~= *include;
                include.includedBy[this] = true;

                // tell all files that depend on this one that the include has been added
                includeAdded(origin, this, *include);

                // now (after includeAdded) we can add a reference between this file and the included one
                //say("adding include-reference from %s to %s", this, *include);
                addReference(origin, *include);
            }
            if (g_print_deps && includes) say("%s includes=%s", this, includes);

            // totally important to touch includes AFTER we know what all of them are
            foreach (include; includes) {
                include.touch;
            }
        }
    }


    // An include has been added from includer (which we depend on) to included.
    // Specialisations of File override this to infer additional depends.
    void includeAdded(ref Origin origin, File includer, File included) {
        //say("File.includeAdded called on %s with %s including %s", this, includer, included);
        foreach (depend; dependedBy.keys()) {
            //say("  passing on to %s", depend);
            depend.includeAdded(origin, includer, included);
        }
    }


    // This file's action is about to be issued, and this is the last chance to
    // augment its action. Specialisation should override this method if augmentation
    // is required. Return true if dependencies were added.
    bool augmentAction() {
        return false;
    }


    // This file has been touched - work out if its action should be issued
    // or if it is now clean, transiting to affected items if this becomes clean.
    // NOTE - nothing can become clean until AFTER all activation has been done by the planner.
    final void touch() {
        if (clean) return;
        if (g_print_details) say("touching %s", path);
        long newest;

        if (activated && action && !action.issued) {
            // this items action may need to be issued
            //say("activated file %s touched", this);

            foreach (depend; action.depends) {
                if (!depend.clean) {
                    if (g_print_details) say("%s waiting for %s to become clean", path, depend.path);
                    return;
                }
                if (newest < depend.includeModTime) {
                    newest = depend.includeModTime;
                    youngestDepend = depend;
                }
            }
            // all files this one depends on are clean

            // give this file a chance to augment its action
            if (augmentAction()) {
                // dependency added - touch this file again to re-check dependencies
                touch;
                return;
            }
            else {
                // no dependencies were added, so we can issue the action now

                if (modTime < newest) {
                    // buildable and out of date - issue action to worker
                    if (g_print_details) {
                        say("%s is out of date with mod_time ", this, modTime);
                        File other      = youngestDepend;
                        File prevOther = this;
                        while (other && other.includeModTime > prevOther.modTime) {
                            say("  %s mod_time %s (younger by %s)",
                                other,
                                other.includeModTime,
                                other.includeModTime - modTime);
                            other = other.youngestDepend;
                        }
                    }
                    action.issue;
                    return;
                }
                else {
                    // already up to date - no need for building
                    if (g_print_details) say("%s is up to date", path);
                    action = null;
                    outstanding.remove(this);
                }
            }
        }

        if (action)   return;
        errorUnless(modTime > 0, Origin(path, 1), "%s (%s) is up to date with zero mod_time!", path, trail);
        // This file is up to date

        // Scan for includes, possibly becoming clean in the process
        if (!scanned) scan;
        if (clean)    return;

        // Find out if includes are clean and what our effective mod_time is
        newest = modTime;
        foreach (include; includes) {
            if (!include.clean) {
                return;
            }
            if (newest < include.includeModTime) {
                newest = include.includeModTime;
                youngestInclude = include;
            }
        }
        includeModTime = newest;
        if (g_print_details) {
            say("%s is clean with effective mod_time %s", this, includeModTime);
            File other      = youngestInclude;
            File prevOther = this;
            while (other && other.includeModTime > prevOther.modTime) {
                say("  %s mod_time %s (younger by %s)",
                    other,
                    other.includeModTime,
                    other.includeModTime - prevOther.modTime);
                other = other.youngestInclude;
            }
        }
        // All includes are clean, so we are too

        clean = true;

        // touch everything that includes or depends on this
        foreach (other; includedBy.byKey()) {
            other.touch;
        }
        foreach (other; dependedBy.byKey()) {
            if (other.activated) other.touch;
        }
    }
}


//
// Obj - an object file built from a source file
//
final class Obj : File {
    static Obj[File] bySource; // object files by the source file they are built from

    this(ref Origin origin, File source) {

        string name_ = setExtension(source.name, "o");
        string path_ = prospectivePath("obj", source.parent, name_);

        super(origin, source.parent, name_, Privacy.PUBLIC, path_, false, true);

        errorUnless(source !in bySource, origin,
                    "source file %s already used to build an object file", source);
        bySource[source] = this;

        string actionName;
        string actionCommand;

        switch (extension(source.name)) {
        case ".cc":
        case ".cpp":
            actionName    = format("%-15s %s", "C++", path);
            actionCommand = format("g++ -c %s -DTRACE_DOMAIN=\\\"%s\\\"",
                                   getOption("C++FLAGS"), source.path);
            foreach (dir; split(getOption("HEADERS"))) {
                actionCommand ~= format(" -isystem %s", dir);
            }
            actionCommand ~= " -iquote src -iquote obj -iquote .";
            actionCommand ~= format(" -o %s %s", path, source.path);
            break;
        case ".c":
            actionName    = format("%-15s %s", "C", path);
            actionCommand = format("gcc -c %s -DTRACE_DOMAIN=\\\"%s\\\"",
                                   getOption("CCFLAGS"), source.path);
            foreach (dir; split(getOption("HEADERS"))) {
                actionCommand ~= format(" -isystem %s", dir);
            }
            actionCommand ~= " -iquote src -iquote obj -iquote .";
            actionCommand ~= format(" -o %s %s", path, source.path);
            break;
        case ".d":
            actionName    = format("%-15s %s", "D", path);
            actionCommand = format("dmd -c %s", getOption("DFLAGS"));
            foreach (dir; split(getOption("IMPORTS"))) {
                actionCommand ~= format(" -I%s", dir);
            }
            actionCommand ~= " -Isrc -Iobj";
            actionCommand ~= format(" -of%s %s", path, source.path);
            break;
        default:
            error(origin, "Unsupported source file extension %s", extension(source.name));
        }

        action = new Action(origin, actionName, actionCommand, [this], [source]);
        addReference(origin, source);
    }
}


//
// Binary - a binary file incorporating object files and 'owning' source files.
//
abstract class Binary : File {
    static Binary[File] byContent; // binaries by the header and body files they 'contain'

    struct Source {
        string  name;
        Privacy privacy;
    }

    bool         isD;
    File[]       objs;
    File[]       headers;
    bool[SysLib] reqSysLibs;
    bool[Binary] reqBinaries;


    // create a binary using files from this package.
    // All the sources themselves may be already-known built files,
    // but can't already be used by another Binary.
    this(ref Origin origin, Pkg pkg, string name_, string path_, string[] requires) {
        super(origin, pkg, name_, Privacy.PUBLIC, path_, false, true);

        // required system libraries
        foreach (req; requires) {
            if (req !in SysLib.byName) {
                new SysLib(req);
            }
            SysLib lib = SysLib.byName[req];
            if (lib !in reqSysLibs) {
                reqSysLibs[lib] = true;
            }
        }
    }

    // Add sources to this Binary. Adding them after construction allows generated source
    // files to be children of the Binary.
    void addMySources(ref Origin origin, Source[] sources) {

        bool isScannable(string extension) {
            return
                extension == ".c"  ||
                extension == ".cc" ||
                extension == ".h"  ||
                extension == ".d";
        }

        errorUnless(sources.length > 0, origin, "binary must have at least one source file");
        foreach (source; sources) {
            string ext = extension(source.name);
            File sourceFile = File.addSource(origin, this, source.name, source.privacy, isScannable(ext));
            byContent[sourceFile] = this;

            if (ext == ".d" || ext == ".cc" || ext == ".c") {
                // a source file that we generate an Obj from
                objs ~= new Obj(origin, sourceFile);

                if (ext == ".d") {
                    isD = true;
                }
            }
            else if (ext == ".ipc") {
                // an inter-process-communication file from which we generate a header file

                // make sure the ipc-compiler has already been defined
                string compilerPath = buildPath("dist", "bin", "ipc-compiler");
                File* compiler = compilerPath in File.byPath;
                errorUnless(compiler !is null,
                            origin,
                            "Cannot cook an ipc files before creating the compiler (%s)",
                            compilerPath);

                // build a header file from this ipc file
                string destPath = buildPath("obj", parent.trail, stripExtension(source.name) ~ ".h");
                File dest = new File(origin, this, baseName(destPath), source.privacy, destPath, true, true);
                dest.action = new Action(origin,
                                         format("%-15s %s", "cook-ipc", sourceFile.path),
                                         format("ipc-compiler %s %s", sourceFile.path, dest.path),
                                         [dest],
                                         [sourceFile, *compiler]);
                byContent[dest] = this;
                headers ~= dest;
            }
            else {
                headers ~= sourceFile;
            }
        }
    }

    override void includeAdded(ref Origin origin, File includer, File included) {
        // A file we depend on (includer) has included another file (included).
        // If this means that this 'needs' another Binary, remember the fact and
        // also add a dependency on that other Binary. Note that the dependency
        // is often not 'real' (a StaticLib doesn't actually depend on other StaticLibs),
        // but it is a very useful simplification when working out which libraries an
        // Exe depends on.
        //say("%s: %s includes %s", this.path, includer.path, included.path);
        if (includer in byContent && byContent[includer] is this) {
            Binary * container = included in byContent;
            errorUnless(container !is null, origin, "included file is not contained in a library");
            if (*container !is this && *container !in reqBinaries) {

                // we require the container of the included file
                reqBinaries[*container] = true;

                // add a dependancy and a reference
                addReference(origin, *container, format(" because %s includes %s", includer.path, included.path));
                action.addDependency(*container);
                //say("%s requires %s", this.path, container.path);
            }
        }
    }
}


//
// StaticLib - a static library.
//
final class StaticLib : Binary {

    string uniqueName;

    this(ref Origin origin, Pkg pkg, string name_, string[] requires) {
        uniqueName = std.array.replace(buildPath(pkg.trail, name_), dirSeparator, "-") ~ "-s";
        if (name_ == pkg.name) uniqueName = std.array.replace(pkg.trail, dirSeparator, "-") ~ "-s";
        string _path = buildPath("obj", format("lib%s.a", uniqueName));
        super(origin, pkg, name_, _path, requires);
    }

    void addSources(ref Origin origin, string[] publicSources, string[] protectedSources) {
        Source[] sources;
        foreach (name; protectedSources) {
            sources ~= Source(name, Privacy.SEMI_PROTECTED);
        }
        foreach (name; publicSources) {
            sources ~= Source(name, Privacy.PUBLIC);
        }
        addMySources(origin, sources);

        // action
        string command;
        if (objs.length) {
            command = format("rm -f %s; ar csr %s", path, path);
            foreach (obj; objs) {
                command ~= format(" %s", obj.path);
            }
        }
        else {
            command = format("rm -f %s; echo dummy > %s", path, path);
        }
        action = new Action(origin, format("%-15s %s", "StaticLib", path), command, [this], objs ~ headers);
    }
}


//
// DynamicLib - a dynamic library. Contains all of the object files
// from a number of specified StaticLibs. If defined prior to an Exe, the Exe will
// link with the DynamicLib instead of those StaticLibs.
//
// Any StaticLibs required by the incorporated StaticLibs must also be incorporated
// into DynamicLibs.
//
// The static lib names are relative to pkg, and therefore only descendants of the DynamicLib's
// parent can be incorporated.
//
final class DynamicLib : File {
    static DynamicLib[StaticLib] byContent; // dynamic libs by the static libs they 'contain'
    Origin origin;
    bool   augmented;
    string uniqueName;

    StaticLib[] staticLibs;

    this(ref Origin origin_, Pkg pkg, string name_, string[] staticTrails, bool isPlugin) {
        origin = origin_;

        uniqueName = std.array.replace(buildPath(pkg.trail, name_), "/", "-");
        if (name_ == pkg.name) uniqueName = std.array.replace(pkg.trail, dirSeparator, "-");
        string _path = buildPath("dist", "lib", format("lib%s.so", uniqueName));
        if (isPlugin) _path = buildPath("dist", "lib", "plugins", format("lib%s.so", uniqueName));

        super(origin, pkg, name_ ~ "-dynamic", Privacy.PUBLIC, _path, false, true);

        foreach (trail; staticTrails) {
            string trail1 = buildPath(pkg.trail, trail, baseName(trail));
            string trail2 = buildPath(pkg.trail, trail);
            Node* node = trail1 in Node.byTrail;
            if (node is null || cast(StaticLib*) node is null) {
                node = trail2 in Node.byTrail;
                if (node is null || cast(StaticLib*) node is null) {
                    error(origin,
                          "Unknown static-lib %s, looked for with trails %s and %s",
                          trail, trail1, trail2);
                }
            }
            StaticLib* staticLib = cast(StaticLib*) node;
            errorUnless(*staticLib !in byContent, origin,
                        "static lib %s already used by dynamic lib %s",
                        *staticLib, byContent[*staticLib]);
            addReference(origin, *staticLib);
            staticLibs ~= *staticLib;
            byContent[*staticLib] = this;
        }
        errorUnless(staticLibs.length > 0, origin, "dynamic-lib must have at least one static-lib");

        // action
        bool[SysLib] gotSysLibs;
        string actionName = format("%-15s %s", "DynamicLib", path);
        string command = format("g++ -shared %s -o %s", getOption("LINKFLAGS"), path);
        File[] depLibs;
        foreach (staticLib; staticLibs) {
            depLibs ~= staticLib;
            foreach (obj; staticLib.objs) {
                command ~= format(" %s", obj.path);
            }
        }

        action = new Action(origin, actionName, command, [cast(File)this], depLibs);
    }


    // Called just before our action is issued.
    // Verify that all the StaticLibs we now know that we depend on are contained by this or
    // another earlier-defined-than-this DynamicLib.
    // Add any required SysLibs to our action.
    override bool augmentAction() {
        if (augmented) return false;
        augmented = true;

        bool[StaticLib] doneStaticLibs;
        bool[SysLib]    gotSysLibs;
        SysLib[]        sysLibs;
        bool            added;

        //say("augmenting action for DynamicLib %s", name);
        void accumulate(StaticLib lib) {
            //say("  accumulating %s", lib.name);
            foreach (other; lib.reqBinaries.keys) {
                StaticLib next = cast(StaticLib) other;
                assert(next !is null);
                if (next !in doneStaticLibs) {
                    accumulate(next);
                }
            }
            if (lib !in doneStaticLibs) {
                doneStaticLibs[lib] = true;
                errorUnless(lib.objs.length == 0 ||
                            (lib in byContent && byContent[lib].number <= number),
                            origin,
                            "dynamic-lib %s requires static-lib %s (%s) which "
                            "is not contained in a pre-defined dynamic-lib",
                            name, lib.trail, lib.path);
                foreach (sys; lib.reqSysLibs.keys) {
                    if (sys !in gotSysLibs) {
                        //say("  new required SysLib %s", sys.name);
                        gotSysLibs[sys] = true;
                        sysLibs ~= sys;
                        added = true;
                    }
                }
            }
        }
        foreach (lib; staticLibs) {
            accumulate(lib);
        }

        string augmentation;
        foreach (sys; sysLibs) {
            augmentation ~= format(" -l%s", sys.name);
        }
        //say("augmentation is %s", augmentation);
        action.augment(augmentation);
        return added;
    }
}


//
// Exe - An executable file
//
final class Exe : Binary {

    bool   augmented;
    string desc;
    string src;
    string dest;

    // create an executable using files from this package, linking to libraries
    // that contain any included header files, and any required system libraries.
    // Note that any system libraries required by inferred local libraries are
    // automatically linked to.
    this(ref Origin origin, Pkg pkg, string kind, string name_, string[] requires) {
        // interpret kind
        switch (kind) {
            case "dist-util": desc = "DistUtil"; src = "util";  dest = buildPath("dist", "bin", name_);     break;
            case "priv-util": desc = "PrivUtil"; src = "util";  dest = buildPath("priv", pkg.trail, name_); break;
            case "test-util": desc = "TestExe";  src = "test";  dest = buildPath("priv", pkg.trail, name_); break;
            default: assert(0, "invalid Exe kind " ~ kind);
        }

        super(origin, pkg, name_ ~ "-exe", dest, requires);

        if (kind == "test-util") {
            File test   = new File(origin, pkg, name ~ "-result",
                                   Privacy.PRIVATE, dest ~ "-passed", false, true);
            test.action = new Action(origin,
                                     format("%-15s %s", "TestResult", test.path),
                                     format("./test %s", dest),
                                     [test],
                                     [this]);
        }
    }


    void addSources(ref Origin origin, string[] sources) {
        errorUnless(sources.length > 0, origin, "An exe must have at least one source file");
        Source[] _sources;
        foreach (source; sources) {
            _sources ~= Source(buildPath(src, source), Privacy.PROTECTED);
        }
        addMySources(origin, _sources);

        // exe action
        string command;
        string ext = extension(sources[0]);
        switch (ext) {
        case ".cc":
        case ".c":
            command = format("%s %s -o %s",
                             ext == "c" ? "gcc" : "g++",
                             getOption("LINKFLAGS"), dest);
            foreach (dir; split(getOption("HEADERS"))) {
                command ~= format(" -isystem %s", dir);
            }
            foreach (obj; objs) {
                command ~= format(" %s", obj.path);
            }
            command ~= format(" -L%s", buildPath("dist", "lib"));
            command ~= format(" -L%s", buildPath("dist", "lib", "plugins"));
            command ~= format(" -Lobj");
            break;
        case ".d":
            command = format("dmd %s -of%s ", getOption("DLINKFLAGS"), path);
            foreach (obj; objs) {
                command ~= format(" %s", obj.path);
            }
            command ~= format(" -L-Lobj");
            break;
        default:
            error(origin, "Unsupported source file extension %s in %s", extension(sources[0]), path);
        }

        action = new Action(origin, format("%-15s %s", desc, dest), command, [this], objs ~ headers);
    }


    // Called just before our action is issued - augment the action's command string
    // with the library dependencies that we should now know about via includeAdded().
    // Return true if dependencies were added.
    override bool augmentAction() {
        if (augmented) return false;
        augmented = true;
        bool added = false;
        //say("augmenting %s's action command", this);

        // binaries we require, with most fundamental first
        bool[DynamicLib] gotDynamicLibs;
        bool[StaticLib]  gotStaticLibs;
        bool[SysLib]     gotSysLibs;
        DynamicLib[]     dynamicLibs;
        StaticLib[]      staticLibs;
        SysLib[]         sysLibs;

        // accumulate the libraries needed
        void accumulate(Binary binary) {
            //say("accumulating binary %s", binary.path);
            foreach (other; binary.reqBinaries.keys) {
                StaticLib lib = cast(StaticLib) other;
                assert(lib !is null);
                if (lib !in gotStaticLibs) {
                    accumulate(other);
                }
            }
            if (binary is this) {
                foreach (sys; reqSysLibs.keys) {
                    if (sys !in gotSysLibs) {
                        //say("    using sys-lib %s", sys.name);
                        gotSysLibs[sys] = true;
                        sysLibs ~= sys;
                    }
                }
            }
            else {
                StaticLib lib = cast(StaticLib) binary;
                if (lib !in gotStaticLibs) {
                    //say("  require static-lib %s", lib.uniqueName);
                    gotStaticLibs[lib] = true;
                    foreach (sys; lib.reqSysLibs.keys) {
                        if (sys !in gotSysLibs) {
                            //say("    using sys-lib %s", sys.name);
                            gotSysLibs[sys] = true;
                            sysLibs ~= sys;
                        }
                    }

                    DynamicLib* dynamic = lib in DynamicLib.byContent;
                    if (dynamic !is null && dynamic.number < this.number) {
                        // use the dynamic lib that contains the static lib
                        if (*dynamic !in gotDynamicLibs) {
                            //say("    using dynamic-lib %s to cover %s", dynamic.name, lib.uniqueName);
                            gotDynamicLibs[*dynamic] = true;
                            dynamicLibs ~= *dynamic;
                            action.addDependency(*dynamic);
                            added = true;

                            // we have to also accumulate everything this dynamic lib needs
                            foreach (contained; dynamic.staticLibs) {
                                accumulate(contained);
                            }
                        }
                    }
                    else {
                        // use the static lib
                        //say("    using static-lib %s", lib.uniqueName);
                        staticLibs ~= lib;
                    }
                }
            }
        }
        //say("accumulating required libraries for exe %s", path);
        accumulate(this);

        string extra;
        if (isD) extra = "-L";

        string augmentation;
        foreach (lib; retro(staticLibs)) {
            if (lib.objs.length) {
                augmentation ~= format(" %s-l%s", extra, lib.uniqueName);
            }
        }
        foreach (lib; retro(dynamicLibs)) {
            augmentation ~= format(" %s-l%s", extra, lib.uniqueName);
        }
        foreach (lib; retro(sysLibs)) {
            augmentation ~= format(" %s-l%s", extra, lib.name);
        }
        //say("%s augmentation is '%s'", this, augmentation);

        action.augment(augmentation);
        return added;
    }
}


//
// Mark all built files in child or referred packages as needed
//
int markNeeded(Node given) {
    static bool[Node] done;

    int qty = 0;
    if (given in done) return qty;
    done[given] = true;

    File file = cast(File) given;
    if (file && file.built) {
        // activate this file
        if (file.action is null) {
            fatal("file %s activated before its action was set", file);
        }
        //say("activating %s", file.path);
        file.activated = true;
        File.allActivated[file] = true;
        File.outstanding[file] = true;
        qty++;
    }

    // recurse into child and referred nodes
    foreach (child; chain(given.children, given.refers)) {
        qty += markNeeded(child);
    }

    if (file) {
        // touch this file to trigger building
        file.touch;
    }

    return qty;
}


//
// Process a Bobfile
//
void processBobfile(string indent, Pkg pkg) {
    static bool[Pkg] processed;
    if (pkg in processed) return;
    processed[pkg] = true;

    if (g_print_rules) say("%sprocessing %s", indent, pkg.bobfile);
    indent ~= "  ";
    foreach (statement; readBobfile(pkg.bobfile.path)) {
        if (g_print_rules) say("%s%s", indent, statement.toString);
        switch (statement.rule) {

            // packages
            case "contain":
                foreach (name; statement.targets) {
                    errorUnless(dirName(name) == ".", statement.origin,
                                "Contained packages have to be top-level");
                    Privacy privacy = privacyOf(statement.origin, statement.arg1);
                    Pkg newPkg = new Pkg(statement.origin, pkg, name, privacy);
                    processBobfile(indent, newPkg);
                }
            break;

            case "refer":
                foreach (trail; statement.targets) {
                    Pkg* other = cast(Pkg*) (trail in Node.byTrail);
                    if (other is null) {
                        // create the referenced package which must be top-level, then refer to it
                        errorUnless(dirName(trail) == ".", statement.origin,
                                    "Previously-unknown referenced package %s has to be top-level", trail);
                        Pkg newPkg = new Pkg(statement.origin, Node.byTrail["root"], trail, Privacy.PUBLIC);
                        processBobfile(indent, newPkg);
                        pkg.addReference(statement.origin, newPkg);
                    }
                    else {
                        // refer to the existing package
                        errorUnless(other !is null, statement.origin,
                                    "Cannot refer to unknown pkg %s", trail);
                        pkg.addReference(statement.origin, *other);
                    }
                }
            break;

            // libraries
            case "static-lib":
            {
                errorUnless(statement.targets.length == 1, statement.origin,
                            "Can only have one static-lib name per rule");
                StaticLib lib = new StaticLib(statement.origin, pkg, statement.targets[0], statement.arg3);
                lib.addSources(statement.origin, statement.arg1, statement.arg2);
            }
            break;

            case "dynamic-lib":
            {
                errorUnless(statement.targets.length == 1, statement.origin,
                            "Can only have one dynamic-lib name per rule");
                new DynamicLib(statement.origin, pkg, statement.targets[0], statement.arg1,
                               statement.arg2.length == 1 && statement.arg2[0] == "plugin");
            }
            break;

            case "dist-util":
            case "priv-util":
            case "test-util":
            {
                errorUnless(statement.targets.length == 1,
                            statement.origin,
                            "Can only have one util/exe name per rule");
                Exe exe = new Exe(statement.origin, pkg, statement.rule, statement.targets[0], statement.arg2);
                exe.addSources(statement.origin, statement.arg1);
            }
            break;

            case "tao-lib":
            {
                //
                // tao-lib  lib-name : idl-sources;    # idl files and gen src in package dir
                //
                errorUnless(statement.targets.length == 1, statement.origin,
                            "Can only have one tao static-lib name per rule");


                //
                // create the static-lib
                //

                auto requires = [
                    "TAO_CosNaming",
                    "TAO_PortableServer",
                    "TAO_DynamicAny",
                    "TAO",
                    "ACE",
                    "rt",
                    "dl"];
                auto lib = new StaticLib(statement.origin, pkg, statement.targets[0], requires);

                //
                // create the idl files and the source files generated from them
                //

                string[] publicNames;
                string[] protectedNames;
                foreach (idlName; statement.arg1) {

                    // idl file
                    File idl = File.addSource(statement.origin, pkg, idlName, Privacy.PROTECTED, false);

                    // paths of all generated files
                    string base = buildPath("obj", pkg.trail, stripExtension(idlName));
                    string[] junk = [
                        base ~ "S_T.cc"];
                    string[] publicPaths = [
                        base ~ "C.h",
                        base ~ "S.h" ];
                    string[] protectedPaths = [
                        base ~ "S_T.inl",
                        base ~ "C.inl",
                        base ~ "S.inl",
                        base ~ "S_T.h",
                        base ~ "C.cc",
                        base ~ "S.cc" ];

                    // generated files
                    File[] files;
                    foreach (path; junk) {
                        files ~= new File(statement.origin, lib, baseName(path),
                                          Privacy.PRIVATE, path, false, true);
                    }
                    foreach (path; publicPaths) {
                        files ~= new File(statement.origin, lib, baseName(path),
                                          Privacy.PUBLIC, path, !(extension(path) == ".inl"), true);
                        publicNames ~= baseName(path);
                    }
                    foreach (path; protectedPaths) {
                        files ~= new File(statement.origin, lib, baseName(path),
                                          Privacy.PROTECTED, path, !(extension(path) == ".inl"), true);
                        protectedNames ~= baseName(path);
                    }

                    // action to generate the files and edit them to be correct
                    string command = format("%s -in -Ce -cs C.cc -ss S.cc -sT S_T.cc", getOption("TAO_IDL"));
                    if (toLower(getOption("GENERATE_EMPTY_SERVANT")) == "true") {
                        command ~= " -GI -GIh \"_impl-rename.h\" -GIs \"_impl-rename.cc\" -GIe \"Impl\" -GIc -GIa";
                    }
                    foreach (d; split(getOption("IDL_HEADERS"))) {
                        command ~= format(" -I%s", d);
                    }
                    command ~= format(" -I%s", dirName(idl.path));
                    command ~= format(" -I%s", dirName(base));
                    command ~= " -o " ~ buildPath("obj", pkg.trail) ~ " " ~ idl.path ~ " &&";
                    command ~=
                        ` sed --in-place --separate` ~
                        ` -e '/#include/s/\"orbsvcs\/\([^\"]*\)\"/\<orbsvcs\/\1\>/'` ~
                        ` -e '/#include/s/\"tao\/\([^\"]*\)\"/\<tao\/\1\>/'` ~
                        ` -e '/#include/s/\"ace\/\([^\"]*\)\"/\<ace\/\1\>/'` ~
                        ` -e '/#include \"/s|#include \"|#include \"` ~ pkg.trail ~ `/|'` ~ // XXX should not do this if included path contains a slash
                        ` -e '/#include .*.cc/d'` ~
                        ` -e 's/[[:space:]]\+$//'`;
                    foreach (file; files) {
                        command ~= " " ~ file.path;
                    }
                    auto action = new Action(statement.origin, format("%-15s %s", statement.rule, idl.path),
                                             command, files, [idl]);

                    foreach (file; files) {
                        file.action = action;
                    }
                }

                // add the generated source files as the sources of the library, completing its definition
                lib.addSources(statement.origin, publicNames, protectedNames);
            }
            break;

            case "dist-data":
            case "test-data":
            case "util-data":
            case "doc-data":
            {
                void dataRule(ref Origin origin, Pkg pkg, string kind, string extra) {
                    string fromPath;
                    string destPath;
                    if (kind == "doc-data") {
                        fromPath = buildPath("src",  pkg.trail, "doc", extra);
                        destPath = buildPath("priv", pkg.trail, "doc", extra);
                    }
                    else if (kind == "test-data") {
                        fromPath = buildPath("src",  pkg.trail, "test", extra);
                        destPath = buildPath("priv", pkg.trail,         extra);
                    }
                    else if (kind == "util-data") {
                        fromPath = buildPath("src",  pkg.trail, "util", extra);
                        destPath = buildPath("priv", pkg.trail,         extra);
                    }
                    else if (kind == "dist-data") {
                        fromPath = buildPath("src",  pkg.trail, "data", extra);
                        destPath = buildPath("dist",            "data", extra);
                    }

                    if (isDir(fromPath)) {
                        // recurse into directory
                        foreach (string path; dirEntries(fromPath, SpanMode.shallow)) {
                            dataRule(origin, pkg, kind, buildPath(extra, path.baseName));
                        }
                    }
                    else {
                        // set up from and dest files, and an action to create the dest.
                        string name = std.array.replace(extra, dirSeparator, "-");
                        File from   = new File(origin, pkg, name ~ "-src",  Privacy.PUBLIC, fromPath, false, false);
                        File dest   = new File(origin, pkg, name ~ "-dest", Privacy.PUBLIC, destPath, false, true);
                        dest.action = new Action(origin,
                                                 format("%-15s %s", "Data", dest.path),
                                                 format("cp %s %s", from.path, dest.path),
                                                 [dest],
                                                 [from]);
                    }
                }

                foreach (name; statement.targets) {
                    dataRule(statement.origin, pkg, statement.rule, name);
                }
            }
            break;

            case "doc":
            {
                foreach (name; statement.targets) {
                    string fromPath = buildPath("src",  pkg.trail, "doc", name ~ ".rst");
                    string destPath = buildPath("priv", pkg.trail, "doc", name ~ ".html");

                    errorUnless(exists(fromPath) && !isDir(fromPath), statement.origin,
                                "%s not found", fromPath);

                    File from   = new File(statement.origin, pkg, name ~ "-rst",
                                           Privacy.PUBLIC, fromPath, false, false);

                    File dest   = new File(statement.origin, pkg, name ~ "-html",
                                           Privacy.PUBLIC, destPath, false, true);

                    dest.action = new Action(statement.origin,
                                             format("%-15s %s", "Doc", dest.path),
                                             format("%s --exit-status=2 %s %s",
                                                    getOption("RST2HTML"), from.path, dest.path),
                                             [dest],
                                             [from]);
                }
            }
            break;

            case "dist-shell":
            case "priv-shell":
            {
                foreach (name; statement.targets) {

                    string fromPath = buildPath("src",  pkg.trail, "util", name);
                    string destPath;
                    if (statement.rule == "dist-shell") {
                        destPath = buildPath("dist", "bin", name);
                    }
                    else {
                        destPath = buildPath("priv", pkg.trail, name);
                    }

                    errorUnless(exists(fromPath) && !isDir(fromPath), statement.origin,
                                "%s not found", fromPath);

                    File from   = new File(statement.origin, pkg, name ~ "-sh",
                                           Privacy.PUBLIC, fromPath, false, false);

                    File dest   = new File(statement.origin, pkg, name,
                                           Privacy.PUBLIC, destPath, false, true);

                    dest.action = new Action(statement.origin,
                                             format("%-15s %s", "Shell", dest.path),
                                             format("cp -f %s %s && chmod +x %s",
                                                    from.path, dest.path, dest.path),
                                             [dest],
                                             [from]);
                }
            }
            break;

        default:
            error(statement.origin, "Unsupported rule '%s'", statement.rule);
        }
    }
}


// remove any files in obj, priv and dist that aren't marked as needed
void cleandirs() {
    void cleanDir(string name) {
        //say("cleaning dir %s, cdw=%s", name, getcwd);
        if (exists(name) && isDir(name)) {
            bool[string] dirs;
            foreach (DirEntry entry; dirEntries(name, SpanMode.depth, false)) {
                //say("  considering %s", entry.name);
                bool isDir = attrIsDir(entry.linkAttributes);

                if (!isDir) {
                    File* file = entry.name in File.byPath;
                    if (file is null || (*file) !in File.allActivated) {
                        say("Removing unwanted file %s", entry.name);
                        std.file.remove(entry.name);
                    }
                    else {
                        // leaving a file in place
                        //say("  keeping activated file %s", file.path);
                        dirs[entry.name.dirName] = true;
                    }
                }
                else {
                    if (entry.name !in dirs) {
                        //say("removing empty dir %s", entry.name);
                        rmdir(entry.name);
                    }
                    else {
                        //say("  keeping non-empty dir %s", entry.name);
                        dirs[entry.name.dirName] = true;
                    }
                }
            }
        }
    }
    cleanDir("obj");
    cleanDir("priv");
    cleanDir("dist");
}


//
// Planner function
//
bool doPlanning(int numJobs, bool printRules, bool printDeps, bool printDetails) {

    // state variables
    size_t       inflight;
    bool[string] workers;
    bool[string] idlers;
    bool         exiting;
    bool         success = true;

    // receive registration message from each worker and remember its name
    while (workers.length < numJobs) {
        receive( (string worker) { workers[worker] = true; idlers[worker] = true; } );
    }

    // local function: an action has completed successfully - update all files built by it
    void actionCompleted(string worker, string action) {
        //say("%s %s succeeded", action, worker);
        --inflight;
        idlers[worker] = true;
        try {
            foreach (file; Action.byName[action].builds) {
                file.updated;
            }
        }
        catch (BailException ex) { exiting = true; success = false; }
    }

    // local function: a worker has terminated - remove it from workers and remember we are exiting
    void workerTerminated(string worker) {
        exiting = true;
        workers.remove(worker);
        //say("%s has terminated - %s workers remaining", worker, workers.length);
    }


    // set up some globals
    readOptions;
    g_print_rules   = printRules;
    g_print_deps    = printDeps;
    g_print_details = printDetails;

    string projectPackage = getOption("PROJECT-PACKAGE");

    int needed;
    try {
        // read the project Bobfile and descend into all those it refers to
        auto root = new Node();
        auto project = new Pkg(Origin(), root, projectPackage, Privacy.PRIVATE);
        processBobfile("", project);

        // mark all files needed by the project Bobfile as needed, priming the action queue
        needed = markNeeded(project);

        cleandirs();
    }
    catch (BailException ex) { exiting = true; success = false; }

    while (workers.length) {

        // give any idle workers something to do
        //say("%s idle workers and %s actions in priority queue", idlers.length, Action.queue.length);
        string[] toilers;
        foreach (idler, dummy; idlers) {

            if (!exiting && !File.outstanding.length) exiting = true;

            Tid tid = std.concurrency.locate(idler);

            if (exiting) {
                // tell idle worker to terminate
                //say("telling %s to terminate", idler);
                send(tid, true);
                toilers ~= idler;
            }
            else if (!Action.queue.empty) {
                // give idle worker an action to perform
                //say("giving %s an action", idler);

                const Action next = Action.queue.front();
                Action.queue.popFront();

                string targets;
                foreach (target; next.builds) {
                    ensureParent(target.path);
                    targets ~= "|" ~ target.path;
                }
                //say("issuing action %s", next.name);
                send(tid, next.name.idup, next.command.idup, targets.idup);
                toilers ~= idler;
                ++inflight;
            }
            else if (!inflight) {
                fatal("nothing to do and no inflight actions");
            }
            else {
                // nothing to do
                //say("nothing to do - waiting for results");
                break;
            }
        }
        foreach (toiler; toilers) idlers.remove(toiler);

        // receive a completion or failure
        receive( (string worker, string action) { actionCompleted(worker, action); },
                 (string worker)                { workerTerminated(worker); } );
    }

    if (!File.outstanding.length && success) {
        say("\n"
            "Total number of files:             %s\n"
            "Number of target files:            %s\n"
            "Number of activated target files:  %s\n"
            "Number of files updated:           %s\n",
            File.byPath.length, File.numBuilt, needed, File.numUpdated);
        return true;
    }
    return false;
}


//-----------------------------------------------------
// Worker
//-----------------------------------------------------

void doWork(bool printActions, uint index, Tid plannerTid) {
    bool success;

    string myName = format("worker-%d", index);
    std.concurrency.register(myName, thisTid);

    void perform(string action, string command, string targets) {
        if (printActions) { say("\n%s\n", command); }
        say("%s", action);

        success = false;
        string results = buildPath(".bob", myName);

        // launch child process to do the action
        string str = command ~ " 2>" ~ results;
        pid_t child = launcher.launch(str);

        // wait for it to complete
        for (;;) {
            int status;
            pid_t wpid = core.sys.posix.sys.wait.waitpid(child, &status, 0);
            if (wpid == -1) {
                // error, possibly a signal - treat as failure
                break;
            }
            else if (wpid == 0) {
                // non-blocking return, which should not happen - treat as failure
                break;
            }
            else if ((status & 0x7f) == 0) {
                // child has terminated - might be success
                success = ((status & 0xff00) >> 8) == 0;
                break;
            }
            else {
                // child state has changed in some way other than termination - treat as failure
                break;
            }
        }
        launcher.completed(child);
        if (!success) {
            bool bailed = launcher.bail;

            // delete built files
            foreach (target; split(targets, "|")) {
                if (exists(target)) {
                    say("  Deleting %s", target);
                    std.file.remove(target);
                }
            }

            // print error message
            if (!bailed) {
                say("\n%s", readText(results));
                say("%s: FAILED\n%s", action, command);
                fatal("Aborting build due to action failure");
            }
            else {
                // just quietly throw
                throw new BailException();
            }
        }
        else {
            // tell planner the action succeeded
            send(plannerTid, myName, action);
        }
    }


    try {
        send(plannerTid, myName);
        bool done;
        while (!done) {
            receive( (string action, string command, string targets) { perform(action, command, targets); },
                     (bool dummy)                                    { done = true; });
        }
    }
    catch (BailException) {}
    catch (Exception ex)  { say("Unexpected exception %s", ex); }

    // tell planner we are terminating
    send(plannerTid, myName);
    //say("%s terminating", myName);
}


//--------------------------------------------------------------------------------------
// main
//
// Assumes that the top-level source packages are all located in a src subdirectory,
// and places build outputs in obj, priv and dist subdirectories.
// The local source paths are necessary to minimise the length of actions,
// and is usually achieved by a configure step setting up sym-links to the
// actual source locations.
//--------------------------------------------------------------------------------------

int main(string[] args) {
    try {
        bool printRules   = false;
        bool printDeps    = false;
        bool printDetails = false;
        bool printActions = false;
        bool help         = false;
        uint numJobs      = 1;

        int returnValue = 0;
        try {
            getopt(args,
                   std.getopt.config.caseSensitive,
                   "rules",        &printRules,
                   "deps",         &printDeps,
                   "details",      &printDetails,
                   "actions",      &printActions,
                   "jobs|j",       &numJobs,
                   "help",         &help);
        }
        catch (std.conv.ConvException ex) {
            returnValue = 2;
            say(ex.msg);
        }
        catch (object.Exception ex) {
            returnValue = 2;
            say(ex.msg);
        }

        if (args.length != 1) {
            say("Option processing failed. There are %s unprocessed argument(s): ", args.length - 1);
            foreach (uint i, arg; args[1..args.length]) {
                say("  %s. \"%s\"", i + 1, arg);
            }
            returnValue = 2;
        }
        if (numJobs < 1) {
            returnValue = 2;
            say("Must allow at least one job!");
        }
        if (returnValue != 0 || help) {
            say("Usage:  bob [options]\n"
                "  --rules          print rules\n"
                "  --deps           print dependencies\n"
                "  --actions        print actions\n"
                "  --details        print heaps of details\n"
                "  --jobs=VALUE     maximum number of simultaneous actions\n"
                "  --help           show this message\n"
                "target is everything contained in the project Bobfile and anything referred to.");
            return returnValue;
        }

        if (printDetails) {
            printActions = true;
            printDeps = true;
        }


        // spawn the workers
        foreach (uint i; 0..numJobs) {
            //say("spawning worker %s", i);
            spawn(&doWork, printActions, i, thisTid);
        }

        // build everything
        returnValue = doPlanning(numJobs, printRules, printDeps, printDetails) ? 0 : 1;

        return returnValue;
    }
    catch (Exception ex) {
        say("got unexpected exception %s", ex);
        return 1;
    }
}