projects/dil: trunk/src/dil/Token.d annotate

annotate trunk/src/dil/Token.d @ 502:4e14cd1b24da

Refactored code and added modules related to tabulated Identifiers. Rearranged members of struct Identifier and added new member ID identID. Moved idTableLookup to module dil.IdTable. Renamed module TokenIDs to TokensEnum. Added member Identifier* ident to struct Token. Changed string switchtes in Parser to integer switches using enum ID.

author	Aziz K?ksal <aziz.koeksal@gmail.com>
date	Tue, 11 Dec 2007 14:19:30 +0100
parents	41b7f9e439bd
children	996041463028

rev	line source
0 8ba2570de175 Initial import. aziz parents: diff changeset	1 /++
8ba2570de175 Initial import. aziz parents: diff changeset	2 Author: Aziz Köksal
249 32d354584b28 - Upgraded license notices to GPL3. aziz parents: 239 diff changeset	3 License: GPL3
0 8ba2570de175 Initial import. aziz parents: diff changeset	4 +/
326 4a7359b88c11 - Added package 'dil' to module declarations. aziz parents: 325 diff changeset	5 module dil.Token;
490 47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	6 import dil.Location;
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 500 diff changeset	7 import dil.Identifier;
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	8 import tango.stdc.stdlib : malloc, free;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	9 import tango.core.Exception;
490 47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	10 import common;
0 8ba2570de175 Initial import. aziz parents: diff changeset	11
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 500 diff changeset	12 public import dil.TokensEnum;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	13
500 41b7f9e439bd Moved enum TOK to new module dil.TokenIDs. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	14 /++
41b7f9e439bd Moved enum TOK to new module dil.TokenIDs. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	15 A Token is a sequence of characters formed by the lexical analyzer.
41b7f9e439bd Moved enum TOK to new module dil.TokenIDs. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	16 +/
0 8ba2570de175 Initial import. aziz parents: diff changeset	17 struct Token
8ba2570de175 Initial import. aziz parents: diff changeset	18 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	19 TOK type; /// The type of the token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	20 /// Pointers to the next and previous tokens (doubly-linked list.)
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	21 Token* next, prev;
131 ce636f3981cc - Removed TOK.Number. aziz parents: 107 diff changeset	22
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	23 char* ws; /// Start of whitespace characters before token. Null if no WS.
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	24 char* start; /// Start of token in source text.
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	25 char* end; /// Points one past the end of token in source text.
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	26
0 8ba2570de175 Initial import. aziz parents: diff changeset	27 union
8ba2570de175 Initial import. aziz parents: diff changeset	28 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	29 /// For newline tokens.
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	30 struct
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	31 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	32 char[] filePath;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	33 uint lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	34 uint lineNum_hline;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 314 diff changeset	35 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	36 /// For #line tokens.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	37 struct
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	38 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	39 Token* tokLineNum; /// #line number
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	40 Token* tokLineFilespec; /// #line number filespec
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	41 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	42 /// For string tokens.
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 314 diff changeset	43 struct
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 314 diff changeset	44 {
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	45 string str;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	46 char pf; /// Postfix 'c', 'w', 'd' or 0 for none.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	47 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	48 Token* tok_str; /// Points to the contents of a token string stored as a
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	49 /// doubly linked list. The last token is always '}' or
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	50 /// EOF in case end of source text is "q{" EOF.
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	51 }
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 500 diff changeset	52 Identifier* ident;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	53 dchar dchar_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	54 long long_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	55 ulong ulong_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	56 int int_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	57 uint uint_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	58 float float_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	59 double double_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	60 real real_;
0 8ba2570de175 Initial import. aziz parents: diff changeset	61 }
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	62
107 722c05bbd5eb - Implemented parseEnumDeclaration() and added class EnumDeclaration. aziz parents: 97 diff changeset	63 alias srcText identifier;
722c05bbd5eb - Implemented parseEnumDeclaration() and added class EnumDeclaration. aziz parents: 97 diff changeset	64
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	65 /// Returns the text of the token.
65 6c21ae79fbb3 - Renamed function Token.span to Token.srcText. aziz parents: 62 diff changeset	66 string srcText()
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	67 {
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	68 assert(start && end);
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	69 return start[0 .. end - start];
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	70 }
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	71
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	72 /// Returns the preceding whitespace of the token.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	73 string wsChars()
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	74 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	75 assert(ws && start);
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	76 return ws[0 .. start - ws];
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	77 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	78
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	79 /// Find next non-whitespace token. Returns 'this' token if the next token is TOK.EOF or null.
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	80 Token* nextNWS()
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	81 out(token)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	82 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	83 assert(token !is null);
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	84 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	85 body
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	86 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	87 auto token = next;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	88 while (token !is null && token.isWhitespace)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	89 token = token.next;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	90 if (token is null \|\| token.type == TOK.EOF)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	91 return this;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	92 return token;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	93 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	94
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	95 /// Find previous non-whitespace token. Returns 'this' token if the previous token is TOK.HEAD or null.
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	96 Token* prevNWS()
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	97 out(token)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	98 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	99 assert(token !is null);
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	100 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	101 body
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	102 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	103 auto token = prev;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	104 while (token !is null && token.isWhitespace)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	105 token = token.prev;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	106 if (token is null \|\| token.type == TOK.HEAD)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	107 return this;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	108 return token;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	109 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	110
210 b7bde6583d3e - Made toString() static. aziz parents: 208 diff changeset	111 static string toString(TOK tok)
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	112 {
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	113 return tokToString[tok];
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	114 }
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	115
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	116 /++
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	117 Returns true if this is a token that can have newlines in it.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	118 These can be block and nested comments and any string literal
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	119 except for escape string literals.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	120 +/
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	121 bool isMultiline()
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	122 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	123 return type == TOK.String && start[0] != '\\' \|\|
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	124 type == TOK.Comment && start[1] != '/';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	125 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	126
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	127 /// Returns true if this is a keyword token.
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	128 bool isKeyword()
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	129 {
163 f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	130 return KeywordsBegin <= type && type <= KeywordsEnd;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	131 }
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	132
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	133 /// Returns true if this is a whitespace token.
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	134 bool isWhitespace()
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	135 {
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	136 return !!(type & TOK.Whitespace);
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	137 }
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	138
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	139 /// Returns true if this is a special token.
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	140 bool isSpecialToken()
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	141 {
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	142 return SpecialTokensBegin <= type && type <= SpecialTokensEnd;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	143 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	144
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	145 version(D2)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	146 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	147 /// Returns true if this is a token string literal.
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	148 bool isTokenStringLiteral()
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	149 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	150 return type == TOK.String && tok_str !is null;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	151 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	152 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	153
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	154 /// Returns true if this token starts a DeclarationDefinition.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	155 bool isDeclDefStart()
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	156 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	157 return isDeclDefStartToken(type);
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	158 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	159
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	160 /// Returns true if this token starts a Statement.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	161 bool isStatementStart()
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	162 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	163 return isStatementStartToken(type);
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	164 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	165
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	166 /// Returns true if this token starts an AsmInstruction.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	167 bool isAsmInstructionStart()
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	168 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	169 return isAsmInstructionStartToken(type);
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	170 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	171
163 f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	172 int opEquals(TOK type2)
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	173 {
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	174 return type == type2;
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	175 }
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	176
490 47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	177 import dil.LexerFuncs;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	178 /// Returns the Location of this token.
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	179 Location getLocation()
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	180 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	181 auto search_t = this.prev;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	182 // Find previous newline token.
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	183 while (search_t.type != TOK.Newline)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	184 search_t = search_t.prev;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	185 auto filePath = search_t.filePath;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	186 auto lineNum = search_t.lineNum - search_t.lineNum_hline;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	187 auto lineBegin = search_t.end;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	188 // Determine actual line begin and line number.
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	189 while (1)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	190 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	191 search_t = search_t.next;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	192 if (search_t == this)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	193 break;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	194 // Multiline tokens must be rescanned for newlines.
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	195 if (search_t.isMultiline)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	196 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	197 auto p = search_t.start, end = search_t.end;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	198 while (p != end)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	199 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	200 if (scanNewline(p) == '\n')
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	201 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	202 lineBegin = p;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	203 ++lineNum;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	204 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	205 else
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	206 ++p;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	207 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	208 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	209 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	210 return new Location(filePath, lineNum, lineBegin, this.start);
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	211 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	212
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	213 uint lineCount()
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	214 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	215 uint count = 1;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	216 if (this.isMultiline)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	217 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	218 auto p = this.start, end = this.end;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	219 while (p != end)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	220 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	221 if (scanNewline(p) == '\n')
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	222 ++count;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	223 else
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	224 ++p;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	225 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	226 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	227 return count;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	228 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	229
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	230 /// Return the source text enclosed by the left and right token.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	231 static char[] textSpan(Token* left, Token* right)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	232 {
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	233 assert(left.end <= right.start \|\| left is right );
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	234 return left.start[0 .. right.end - left.start];
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	235 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	236
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	237 new(size_t size)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	238 {
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	239 void* p = malloc(size);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	240 if (p is null)
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	241 throw new OutOfMemoryException(__FILE__, __LINE__);
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	242 cast(Token)p = Token.init;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	243 return p;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	244 }
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	245
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	246 delete(void* p)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	247 {
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	248 auto token = cast(Token*)p;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	249 if (token)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	250 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	251 if(token.type == TOK.HashLine)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	252 token.destructHashLineToken();
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	253 else
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	254 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	255 version(D2)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	256 if (token.isTokenStringLiteral)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	257 token.destructTokenStringLiteral();
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	258 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	259 }
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	260 free(p);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	261 }
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	262
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	263 void destructHashLineToken()
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	264 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	265 assert(type == TOK.HashLine);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	266 delete tokLineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	267 delete tokLineFilespec;
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	268 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	269
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	270 version(D2)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	271 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	272 void destructTokenStringLiteral()
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	273 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	274 assert(type == TOK.String);
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	275 assert(start && *start == 'q' && start[1] == '{');
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	276 assert(tok_str !is null);
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	277 auto tok_it = tok_str;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	278 auto tok_del = tok_str;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	279 while (tok_it && tok_it.type != TOK.EOF)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	280 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	281 tok_it = tok_it.next;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	282 assert(tok_del && tok_del.type != TOK.EOF);
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	283 delete tok_del;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	284 tok_del = tok_it;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	285 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	286 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	287 }
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	288 }
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	289
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	290 /++
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	291 Not used at the moment. Could be useful if more
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	292 info is needed about the location of nodes/tokens.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	293 +/
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	294 struct NewlineInfo
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	295 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	296 char[] oriPath; /// Original path to the source text.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	297 char[] setPath; /// Path set by #line.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	298 uint oriLineNum; /// Actual line number in the source text.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	299 uint setLineNum; /// Delta line number set by #line.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	300 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	301
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	302 /// Returns true if this token starts a DeclarationDefinition.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	303 bool isDeclDefStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	304 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	305 switch (tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	306 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	307 alias TOK T;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	308 case T.Align, T.Pragma, T.Export, T.Private, T.Package, T.Protected,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	309 T.Public, T.Extern, T.Deprecated, T.Override, T.Abstract,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	310 T.Synchronized, T.Static, T.Final, T.Const, T.Invariant/D 2.0/,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	311 T.Auto, T.Scope, T.Alias, T.Typedef, T.Import, T.Enum, T.Class,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	312 T.Interface, T.Struct, T.Union, T.This, T.Tilde, T.Unittest, T.Debug,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	313 T.Version, T.Template, T.New, T.Delete, T.Mixin, T.Semicolon,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	314 T.Identifier, T.Dot, T.Typeof,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	315 T.Char, T.Wchar, T.Dchar, T.Bool,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	316 T.Byte, T.Ubyte, T.Short, T.Ushort,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	317 T.Int, T.Uint, T.Long, T.Ulong,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	318 T.Float, T.Double, T.Real,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	319 T.Ifloat, T.Idouble, T.Ireal,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	320 T.Cfloat, T.Cdouble, T.Creal, T.Void:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	321 return true;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	322 default:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	323 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	324 return false;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	325 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	326
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	327 /// Returns true if this token starts a Statement.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	328 bool isStatementStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	329 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	330 switch (tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	331 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	332 alias TOK T;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	333 case T.Align, T.Extern, T.Final, T.Const, T.Auto, T.Identifier, T.Dot,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	334 T.Typeof, T.If, T.While, T.Do, T.For, T.Foreach, T.Foreach_reverse,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	335 T.Switch, T.Case, T.Default, T.Continue, T.Break, T.Return, T.Goto,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	336 T.With, T.Synchronized, T.Try, T.Throw, T.Scope, T.Volatile, T.Asm,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	337 T.Pragma, T.Mixin, T.Static, T.Debug, T.Version, T.Alias, T.Semicolon,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	338 T.Enum, T.Class, T.Interface, T.Struct, T.Union, T.LBrace, T.Typedef,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	339 T.This, T.Super, T.Null, T.True, T.False, T.Int32, T.Int64, T.Uint32,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	340 T.Uint64, T.Float32, T.Float64, T.Float80, T.Imaginary32,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	341 T.Imaginary64, T.Imaginary80, T.CharLiteral, T.WCharLiteral,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	342 T.DCharLiteral, T.String, T.LBracket, T.Function, T.Delegate,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	343 T.Assert, T.Import, T.Typeid, T.Is, T.LParen, T.Traits/D2.0/,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	344 T.AndBinary, T.PlusPlus, T.MinusMinus, T.Mul,T.Minus, T.Plus, T.Not,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	345 T.Tilde, T.New, T.Delete, T.Cast:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	346 case T.Char, T.Wchar, T.Dchar, T.Bool,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	347 T.Byte, T.Ubyte, T.Short, T.Ushort,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	348 T.Int, T.Uint, T.Long, T.Ulong,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	349 T.Float, T.Double, T.Real,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	350 T.Ifloat, T.Idouble, T.Ireal,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	351 T.Cfloat, T.Cdouble, T.Creal, T.Void:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	352 return true;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	353 default:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	354 if (SpecialTokensBegin <= tok && tok <= SpecialTokensEnd)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	355 return true;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	356 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	357 return false;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	358 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	359
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	360 /// Returns true if this token starts an AsmInstruction.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	361 bool isAsmInstructionStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	362 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	363 switch(tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	364 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	365 alias TOK T;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	366 case T.In, T.Int, T.Out, T.Identifier, T.Align, T.Semicolon:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	367 return true;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	368 default:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	369 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	370 return false;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	371 }

Mercurial > projects > dil

annotate trunk/src/dil/Token.d @ 502:4e14cd1b24da