Mercurial > projects > dil

annotate trunk/src/dil/Token.d @ 507:996041463028

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Removed TOK.WCharLiteral and TOK.DCharLiteral. Removed unnecessary TOK members.
author Aziz K?ksal <aziz.koeksal@gmail.com>
date Wed, 12 Dec 2007 02:41:30 +0100
parents 4e14cd1b24da
children 943ecc9c133a
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
1 /++
8ba2570de175 Initial import.
aziz
parents:
diff changeset
2 Author: Aziz Köksal
249
32d354584b28 - Upgraded license notices to GPL3.
aziz
parents: 239
diff changeset
3 License: GPL3
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
4 +/
326
4a7359b88c11 - Added package 'dil' to module declarations.
aziz
parents: 325
diff changeset
5 module dil.Token;
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
6 import dil.Location;
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 500
diff changeset
7 import dil.Identifier;
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
8 import tango.stdc.stdlib : malloc, free;
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
9 import tango.core.Exception;
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
10 import common;
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
11
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 500
diff changeset
12 public import dil.TokensEnum;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 32
diff changeset
13
500
41b7f9e439bd Moved enum TOK to new module dil.TokenIDs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
14 /++
41b7f9e439bd Moved enum TOK to new module dil.TokenIDs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
15 A Token is a sequence of characters formed by the lexical analyzer.
41b7f9e439bd Moved enum TOK to new module dil.TokenIDs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
16 +/
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
17 struct Token
8ba2570de175 Initial import.
aziz
parents:
diff changeset
18 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
19 TOK type; /// The type of the token.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
20 /// Pointers to the next and previous tokens (doubly-linked list.)
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
21 Token* next, prev;
131
ce636f3981cc - Removed TOK.Number.
aziz
parents: 107
diff changeset
22
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
23 char* ws; /// Start of whitespace characters before token. Null if no WS.
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
24 char* start; /// Start of token in source text.
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
25 char* end; /// Points one past the end of token in source text.
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
26
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
27 union
8ba2570de175 Initial import.
aziz
parents:
diff changeset
28 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
29 /// For newline tokens.
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
30 struct
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
31 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
32 char[] filePath;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
33 uint lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
34 uint lineNum_hline;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 314
diff changeset
35 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
36 /// For #line tokens.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
37 struct
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
38 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
39 Token* tokLineNum; /// #line number
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
40 Token* tokLineFilespec; /// #line number filespec
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
41 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
42 /// For string tokens.
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 314
diff changeset
43 struct
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 314
diff changeset
44 {
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
45 string str;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
46 char pf; /// Postfix 'c', 'w', 'd' or 0 for none.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 372
diff changeset
47 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 372
diff changeset
48 Token* tok_str; /// Points to the contents of a token string stored as a
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 372
diff changeset
49 /// doubly linked list. The last token is always '}' or
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 372
diff changeset
50 /// EOF in case end of source text is "q{" EOF.
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
51 }
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 500
diff changeset
52 Identifier* ident;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 58
diff changeset
53 dchar dchar_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 58
diff changeset
54 long long_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 58
diff changeset
55 ulong ulong_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 58
diff changeset
56 int int_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 58
diff changeset
57 uint uint_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 58
diff changeset
58 float float_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 58
diff changeset
59 double double_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 58
diff changeset
60 real real_;
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
61 }
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
62
107
722c05bbd5eb - Implemented parseEnumDeclaration() and added class EnumDeclaration.
aziz
parents: 97
diff changeset
63 alias srcText identifier;
722c05bbd5eb - Implemented parseEnumDeclaration() and added class EnumDeclaration.
aziz
parents: 97
diff changeset
64
487
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
65 /// Returns the text of the token.
65
6c21ae79fbb3 - Renamed function Token.span to Token.srcText.
aziz
parents: 62
diff changeset
66 string srcText()
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
67 {
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
68 assert(start && end);
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
69 return start[0 .. end - start];
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
70 }
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
71
487
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
72 /// Returns the preceding whitespace of the token.
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
73 string wsChars()
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
74 {
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
75 assert(ws && start);
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
76 return ws[0 .. start - ws];
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
77 }
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
78
359
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
79 /// Find next non-whitespace token. Returns 'this' token if the next token is TOK.EOF or null.
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
80 Token* nextNWS()
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
81 out(token)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
82 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
83 assert(token !is null);
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
84 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
85 body
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
86 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
87 auto token = next;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
88 while (token !is null && token.isWhitespace)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
89 token = token.next;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
90 if (token is null || token.type == TOK.EOF)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
91 return this;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
92 return token;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
93 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
94
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
95 /// Find previous non-whitespace token. Returns 'this' token if the previous token is TOK.HEAD or null.
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
96 Token* prevNWS()
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
97 out(token)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
98 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
99 assert(token !is null);
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
100 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
101 body
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
102 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
103 auto token = prev;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
104 while (token !is null && token.isWhitespace)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
105 token = token.prev;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
106 if (token is null || token.type == TOK.HEAD)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
107 return this;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
108 return token;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
109 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
110
210
b7bde6583d3e - Made toString() static.
aziz
parents: 208
diff changeset
111 static string toString(TOK tok)
208
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
112 {
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
113 return tokToString[tok];
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
114 }
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
115
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
116 /++
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
117 Returns true if this is a token that can have newlines in it.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
118 These can be block and nested comments and any string literal
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
119 except for escape string literals.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
120 +/
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
121 bool isMultiline()
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
122 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
123 return type == TOK.String && start[0] != '\\' ||
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
124 type == TOK.Comment && start[1] != '/';
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
125 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
126
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
127 /// Returns true if this is a keyword token.
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
128 bool isKeyword()
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
129 {
163
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null.
aziz
parents: 131
diff changeset
130 return KeywordsBegin <= type && type <= KeywordsEnd;
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
131 }
208
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
132
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
133 /// Returns true if this is a whitespace token.
314
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such.
aziz
parents: 312
diff changeset
134 bool isWhitespace()
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such.
aziz
parents: 312
diff changeset
135 {
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such.
aziz
parents: 312
diff changeset
136 return !!(type & TOK.Whitespace);
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such.
aziz
parents: 312
diff changeset
137 }
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such.
aziz
parents: 312
diff changeset
138
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
139 /// Returns true if this is a special token.
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 326
diff changeset
140 bool isSpecialToken()
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 326
diff changeset
141 {
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
142 return SpecialTokensBegin <= type && type <= SpecialTokensEnd;
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 326
diff changeset
143 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 326
diff changeset
144
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
145 version(D2)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
146 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
147 /// Returns true if this is a token string literal.
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
148 bool isTokenStringLiteral()
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
149 {
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
150 return type == TOK.String && tok_str !is null;
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
151 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
152 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
153
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
154 /// Returns true if this token starts a DeclarationDefinition.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
155 bool isDeclDefStart()
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
156 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
157 return isDeclDefStartToken(type);
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
158 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
159
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
160 /// Returns true if this token starts a Statement.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
161 bool isStatementStart()
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
162 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
163 return isStatementStartToken(type);
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
164 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
165
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
166 /// Returns true if this token starts an AsmInstruction.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
167 bool isAsmInstructionStart()
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
168 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
169 return isAsmInstructionStartToken(type);
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
170 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
171
163
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null.
aziz
parents: 131
diff changeset
172 int opEquals(TOK type2)
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null.
aziz
parents: 131
diff changeset
173 {
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null.
aziz
parents: 131
diff changeset
174 return type == type2;
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null.
aziz
parents: 131
diff changeset
175 }
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
176
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
177 import dil.LexerFuncs;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
178 /// Returns the Location of this token.
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
179 Location getLocation()
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
180 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
181 auto search_t = this.prev;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
182 // Find previous newline token.
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
183 while (search_t.type != TOK.Newline)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
184 search_t = search_t.prev;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
185 auto filePath = search_t.filePath;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
186 auto lineNum = search_t.lineNum - search_t.lineNum_hline;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
187 auto lineBegin = search_t.end;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
188 // Determine actual line begin and line number.
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
189 while (1)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
190 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
191 search_t = search_t.next;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
192 if (search_t == this)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
193 break;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
194 // Multiline tokens must be rescanned for newlines.
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
195 if (search_t.isMultiline)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
196 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
197 auto p = search_t.start, end = search_t.end;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
198 while (p != end)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
199 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
200 if (scanNewline(p) == '\n')
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
201 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
202 lineBegin = p;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
203 ++lineNum;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
204 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
205 else
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
206 ++p;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
207 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
208 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
209 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
210 return new Location(filePath, lineNum, lineBegin, this.start);
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
211 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
212
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
213 uint lineCount()
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
214 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
215 uint count = 1;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
216 if (this.isMultiline)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
217 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
218 auto p = this.start, end = this.end;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
219 while (p != end)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
220 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
221 if (scanNewline(p) == '\n')
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
222 ++count;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
223 else
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
224 ++p;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
225 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
226 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
227 return count;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
228 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
229
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
230 /// Return the source text enclosed by the left and right token.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
231 static char[] textSpan(Token* left, Token* right)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
232 {
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
233 assert(left.end <= right.start || left is right );
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
234 return left.start[0 .. right.end - left.start];
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
235 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
236
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
237 new(size_t size)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
238 {
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
239 void* p = malloc(size);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
240 if (p is null)
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
241 throw new OutOfMemoryException(__FILE__, __LINE__);
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
242 *cast(Token*)p = Token.init;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
243 return p;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
244 }
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
245
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
246 delete(void* p)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
247 {
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
248 auto token = cast(Token*)p;
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
249 if (token)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
250 {
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
251 if(token.type == TOK.HashLine)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
252 token.destructHashLineToken();
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
253 else
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
254 {
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
255 version(D2)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
256 if (token.isTokenStringLiteral)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
257 token.destructTokenStringLiteral();
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
258 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
259 }
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
260 free(p);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
261 }
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
262
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
263 void destructHashLineToken()
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
264 {
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
265 assert(type == TOK.HashLine);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
266 delete tokLineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
267 delete tokLineFilespec;
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
268 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
269
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
270 version(D2)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
271 {
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
272 void destructTokenStringLiteral()
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
273 {
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
274 assert(type == TOK.String);
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
275 assert(start && *start == 'q' && start[1] == '{');
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
276 assert(tok_str !is null);
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
277 auto tok_it = tok_str;
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
278 auto tok_del = tok_str;
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
279 while (tok_it && tok_it.type != TOK.EOF)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
280 {
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
281 tok_it = tok_it.next;
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
282 assert(tok_del && tok_del.type != TOK.EOF);
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
283 delete tok_del;
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
284 tok_del = tok_it;
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
285 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
286 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
287 }
208
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
288 }
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
289
487
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
290 /++
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
291 Not used at the moment. Could be useful if more
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
292 info is needed about the location of nodes/tokens.
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
293 +/
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
294 struct NewlineInfo
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
295 {
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
296 char[] oriPath; /// Original path to the source text.
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
297 char[] setPath; /// Path set by #line.
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
298 uint oriLineNum; /// Actual line number in the source text.
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
299 uint setLineNum; /// Delta line number set by #line.
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
300 }
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
301
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
302 /// Returns true if this token starts a DeclarationDefinition.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
303 bool isDeclDefStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
304 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
305 switch (tok)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
306 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
307 alias TOK T;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
308 case T.Align, T.Pragma, T.Export, T.Private, T.Package, T.Protected,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
309 T.Public, T.Extern, T.Deprecated, T.Override, T.Abstract,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
310 T.Synchronized, T.Static, T.Final, T.Const, T.Invariant/*D 2.0*/,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
311 T.Auto, T.Scope, T.Alias, T.Typedef, T.Import, T.Enum, T.Class,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
312 T.Interface, T.Struct, T.Union, T.This, T.Tilde, T.Unittest, T.Debug,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
313 T.Version, T.Template, T.New, T.Delete, T.Mixin, T.Semicolon,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
314 T.Identifier, T.Dot, T.Typeof,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
315 T.Char, T.Wchar, T.Dchar, T.Bool,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
316 T.Byte, T.Ubyte, T.Short, T.Ushort,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
317 T.Int, T.Uint, T.Long, T.Ulong,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
318 T.Float, T.Double, T.Real,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
319 T.Ifloat, T.Idouble, T.Ireal,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
320 T.Cfloat, T.Cdouble, T.Creal, T.Void:
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
321 return true;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
322 default:
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
323 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
324 return false;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
325 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
326
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
327 /// Returns true if this token starts a Statement.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
328 bool isStatementStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
329 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
330 switch (tok)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
331 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
332 alias TOK T;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
333 case T.Align, T.Extern, T.Final, T.Const, T.Auto, T.Identifier, T.Dot,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
334 T.Typeof, T.If, T.While, T.Do, T.For, T.Foreach, T.Foreach_reverse,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
335 T.Switch, T.Case, T.Default, T.Continue, T.Break, T.Return, T.Goto,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
336 T.With, T.Synchronized, T.Try, T.Throw, T.Scope, T.Volatile, T.Asm,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
337 T.Pragma, T.Mixin, T.Static, T.Debug, T.Version, T.Alias, T.Semicolon,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
338 T.Enum, T.Class, T.Interface, T.Struct, T.Union, T.LBrace, T.Typedef,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
339 T.This, T.Super, T.Null, T.True, T.False, T.Int32, T.Int64, T.Uint32,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
340 T.Uint64, T.Float32, T.Float64, T.Float80, T.Imaginary32,
507
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 502
diff changeset
341 T.Imaginary64, T.Imaginary80, T.CharLiteral, T.String, T.LBracket,
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 502
diff changeset
342 T.Function, T.Delegate, T.Assert, T.Import, T.Typeid, T.Is, T.LParen,
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 502
diff changeset
343 T.Traits/*D2.0*/, T.AndBinary, T.PlusPlus, T.MinusMinus, T.Mul,
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 502
diff changeset
344 T.Minus, T.Plus, T.Not, T.Tilde, T.New, T.Delete, T.Cast:
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
345 case T.Char, T.Wchar, T.Dchar, T.Bool,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
346 T.Byte, T.Ubyte, T.Short, T.Ushort,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
347 T.Int, T.Uint, T.Long, T.Ulong,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
348 T.Float, T.Double, T.Real,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
349 T.Ifloat, T.Idouble, T.Ireal,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
350 T.Cfloat, T.Cdouble, T.Creal, T.Void:
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
351 return true;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
352 default:
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
353 if (SpecialTokensBegin <= tok && tok <= SpecialTokensEnd)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
354 return true;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
355 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
356 return false;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
357 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
358
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
359 /// Returns true if this token starts an AsmInstruction.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
360 bool isAsmInstructionStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
361 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
362 switch(tok)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
363 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
364 alias TOK T;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
365 case T.In, T.Int, T.Out, T.Identifier, T.Align, T.Semicolon:
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
366 return true;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
367 default:
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
368 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
369 return false;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
370 }