Applied fixes and improvements to the Lexer class. In scanDelimitedStringLiteral: - leading newlines are skipped now. - passed c instead of closing_delim to isUniAlpha(). - the matching string delimiter must follow a newline. - corrected if-statements and an assert(). Added inTokenString member: - in scanTokenStringLiteral() this member is increment at the start and decremented at the end. - in scanSpecialTokenSequence() this member is checked and #line is not evaluated if it is not zero. Added missing break to case 'L' in scanNumber(). In scanReal(): - changed debug switch to a delegate inside an assert(). - removing underscores from buffer more efficiently. Made some other trivial changes and fixes to other modules.

--- a/trunk/src/cmd/Generate.d	Sat Sep 15 17:12:26 2007 +0200
+++ b/trunk/src/cmd/Generate.d	Mon Sep 17 20:35:08 2007 +0200
@@ -12,14 +12,16 @@
 
 enum DocOption
 {
-  Tokens,
+  Empty,
+  Tokens = 1,
   Syntax = 1<<1,
-  HTML = 1<<2,
-  XML = 1<<3
+  HTML   = 1<<2,
+  XML    = 1<<3
 }
 
 void execute(string fileName, DocOption options)
 {
+  assert(options != DocOption.Empty);
   if (options & DocOption.Syntax)
     syntaxToDoc(fileName, Stdout, options);
   else