Mercurial > projects > dil

comparison trunk/src/dil/Token.d @ 390:4d36eea1bbc9

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Refactored Lexer.scan(). Illegal characters are not ignored anymore. They are reported as errors. Added a new member 'ws' to Token. When a token is scanned the lexer sets ws to the leading whitespace or leaves it at null when no whitespace was found. Added Illegal to enum TOK and IllegalCharacter to enum MID. Added localized messages for MID.IllegalCharacter. Adapted code of cmd.Generate to make use of Token.ws.
author Aziz K?ksal <aziz.koeksal@gmail.com>
date Wed, 12 Sep 2007 21:03:41 +0200
parents 6a5fc22cae34
children 33b566df6af4
comparison
equal deleted inserted replaced
389:c4bfceab7246 390:4d36eea1bbc9
16 { 16 {
17 Invalid, 17 Invalid,
18 18
19 /// Flag for whitespace tokens that must be ignored in the parsing phase. 19 /// Flag for whitespace tokens that must be ignored in the parsing phase.
20 Whitespace = 0x8000, 20 Whitespace = 0x8000,
21 Comment = 1 | Whitespace, 21 Illegal = 1 | Whitespace,
22 Shebang = 2 | Whitespace, 22 Comment = 2 | Whitespace,
23 HashLine = 3 | Whitespace, 23 Shebang = 3 | Whitespace,
24 Filespec = 4 | Whitespace, 24 HashLine = 4 | Whitespace,
25 25 Filespec = 5 | Whitespace,
26 Identifier = 5, 26
27 Identifier = 6,
27 String, 28 String,
28 CharLiteral, WCharLiteral, DCharLiteral, 29 CharLiteral, WCharLiteral, DCharLiteral,
29 30
30 // Special tokens 31 // Special tokens
31 FILE, 32 FILE,
119 TOK type; 120 TOK type;
120 // Position pos; 121 // Position pos;
121 122
122 Token* next, prev; 123 Token* next, prev;
123 124
124 char* start; 125 char* ws; /// Start of whitespace characters before token. Null if no WS.
125 char* end; 126 char* start; /// Start of token in source text.
127 char* end; /// Points one past the end of token in source text.
126 128
127 union 129 union
128 { 130 {
129 struct 131 struct
130 { 132 {
234 } 236 }
235 237
236 const string[] tokToString = [ 238 const string[] tokToString = [
237 "Invalid", 239 "Invalid",
238 240
241 "Illegal",
239 "Comment", 242 "Comment",
240 "#! /shebang/", 243 "#! /shebang/",
241 "#line", 244 "#line",
242 `"filespec"`, 245 `"filespec"`,
243 246