Mercurial > projects > dil

annotate trunk/src/dil/lexer/Token.d @ 783:8380fb2c765f

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Added documentation comments.
author Aziz K?ksal <aziz.koeksal@gmail.com>
date Sat, 23 Feb 2008 02:15:41 +0100
parents 6dbbb403fc58
children c24be8d4f6ab
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
1 /++
8ba2570de175 Initial import.
aziz
parents:
diff changeset
2 Author: Aziz Köksal
249
32d354584b28 - Upgraded license notices to GPL3.
aziz
parents: 239
diff changeset
3 License: GPL3
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
4 +/
596
39fac5531b85 Moved dil.Token to dil.lexer.Token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
5 module dil.lexer.Token;
577
9e811db780a6 Moved LexerFuncs.d to package 'lexer'.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 552
diff changeset
6
600
041eae272362 Moved dil.Identifier to dil.lexer.Identifier.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 597
diff changeset
7 import dil.lexer.Identifier;
041eae272362 Moved dil.Identifier to dil.lexer.Identifier.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 597
diff changeset
8 import dil.lexer.Funcs;
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
9 import dil.Location;
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
10 import tango.stdc.stdlib : malloc, free;
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
11 import tango.core.Exception;
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
12 import common;
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
13
597
4d50267f59c9 Moved dil.TokensEnum to dil.lexer.TokensEnum.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 596
diff changeset
14 public import dil.lexer.TokensEnum;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 32
diff changeset
15
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
16 /// A Token is a sequence of characters formed by the lexical analyzer.
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
17 struct Token
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
18 { /// Flags set by the Lexer.
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
19 enum Flags : ushort
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
20 {
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
21 None,
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
22 Whitespace = 1, /// Tokens with this flag are ignored by the Parser.
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
23 }
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
24
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
25 TOK kind; /// The token kind.
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
26 Flags flags; /// The flags of the token.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
27 /// Pointers to the next and previous tokens (doubly-linked list.)
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
28 Token* next, prev;
131
ce636f3981cc - Removed TOK.Number.
aziz
parents: 107
diff changeset
29
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
30 /// Start of whitespace characters before token. Null if no WS.
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
31 /// TODO: remove to save space; can be replaced by 'prev.end'.
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
32 char* ws;
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
33 char* start; /// Start of token in source text.
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
34 char* end; /// Points one past the end of token in source text.
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
35
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
36 /// Data associated with this token.
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
37 /// TODO: move data structures out; use only pointers here to keep Token.sizeof small.
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
38 union
8ba2570de175 Initial import.
aziz
parents:
diff changeset
39 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
40 /// For newline tokens.
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
41 NewlineData newline;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
42 /// For #line tokens.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
43 struct
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
44 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
45 Token* tokLineNum; /// #line number
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
46 Token* tokLineFilespec; /// #line number filespec
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
47 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
48 /// For string tokens.
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 314
diff changeset
49 struct
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 314
diff changeset
50 {
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
51 string str; /// Zero-terminated string.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
52 char pf; /// Postfix 'c', 'w', 'd' or 0 for none.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 372
diff changeset
53 version(D2)
612
c65b11c2074c Added getDocComments() to class Node.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
54 Token* tok_str; /++ Points to the contents of a token string stored as a
c65b11c2074c Added getDocComments() to class Node.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
55 doubly linked list. The last token is always '}' or
c65b11c2074c Added getDocComments() to class Node.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
56 EOF in case end of source text is "q{" EOF.
c65b11c2074c Added getDocComments() to class Node.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
57 +/
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
58 }
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
59 Identifier* ident; /// For keywords and identifiers.
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 58
diff changeset
60 dchar dchar_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 58
diff changeset
61 long long_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 58
diff changeset
62 ulong ulong_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 58
diff changeset
63 int int_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 58
diff changeset
64 uint uint_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 58
diff changeset
65 float float_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 58
diff changeset
66 double double_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 58
diff changeset
67 real real_;
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
68 }
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
69
487
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
70 /// Returns the text of the token.
65
6c21ae79fbb3 - Renamed function Token.span to Token.srcText.
aziz
parents: 62
diff changeset
71 string srcText()
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
72 {
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
73 assert(start && end);
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
74 return start[0 .. end - start];
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
75 }
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
76
487
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
77 /// Returns the preceding whitespace of the token.
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
78 string wsChars()
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
79 {
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
80 assert(ws && start);
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
81 return ws[0 .. start - ws];
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
82 }
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
83
359
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
84 /// Find next non-whitespace token. Returns 'this' token if the next token is TOK.EOF or null.
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
85 Token* nextNWS()
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
86 out(token)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
87 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
88 assert(token !is null);
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
89 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
90 body
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
91 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
92 auto token = next;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
93 while (token !is null && token.isWhitespace)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
94 token = token.next;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
95 if (token is null || token.kind == TOK.EOF)
359
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
96 return this;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
97 return token;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
98 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
99
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
100 /// Find previous non-whitespace token. Returns 'this' token if the previous token is TOK.HEAD or null.
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
101 Token* prevNWS()
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
102 out(token)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
103 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
104 assert(token !is null);
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
105 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
106 body
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
107 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
108 auto token = prev;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
109 while (token !is null && token.isWhitespace)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
110 token = token.prev;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
111 if (token is null || token.kind == TOK.HEAD)
359
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
112 return this;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
113 return token;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
114 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
115
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
116 /// Returns the string for token kind tok.
210
b7bde6583d3e - Made toString() static.
aziz
parents: 208
diff changeset
117 static string toString(TOK tok)
208
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
118 {
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
119 return tokToString[tok];
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
120 }
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
121
747
00f872d949ea Added method scanCommentText() to DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 681
diff changeset
122 /// Adds Flags. Whitespace to this token's flags.
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
123 void setWhitespaceFlag()
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
124 {
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
125 this.flags |= Flags.Whitespace;
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
126 }
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
127
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
128 /// Returns true if this is a token that can have newlines in it.
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
129 /// These can be block and nested comments and any string literal
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
130 /// except for escape string literals.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
131 bool isMultiline()
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
132 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
133 return kind == TOK.String && start[0] != '\\' ||
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
134 kind == TOK.Comment && start[1] != '/';
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
135 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
136
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
137 /// Returns true if this is a keyword token.
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
138 bool isKeyword()
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
139 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
140 return KeywordsBegin <= kind && kind <= KeywordsEnd;
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
141 }
208
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
142
508
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
143 /// Returns true if this is an integral type token.
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
144 bool isIntegralType()
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
145 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
146 return IntegralTypeBegin <= kind && kind <= IntegralTypeEnd;
508
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
147 }
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
148
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
149 /// Returns true if this is a whitespace token.
314
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such.
aziz
parents: 312
diff changeset
150 bool isWhitespace()
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such.
aziz
parents: 312
diff changeset
151 {
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
152 return !!(flags & Flags.Whitespace);
314
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such.
aziz
parents: 312
diff changeset
153 }
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such.
aziz
parents: 312
diff changeset
154
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
155 /// Returns true if this is a special token.
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 326
diff changeset
156 bool isSpecialToken()
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 326
diff changeset
157 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
158 return SpecialTokensBegin <= kind && kind <= SpecialTokensEnd;
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 326
diff changeset
159 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 326
diff changeset
160
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
161 version(D2)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
162 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
163 /// Returns true if this is a token string literal.
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
164 bool isTokenStringLiteral()
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
165 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
166 return kind == TOK.String && tok_str !is null;
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
167 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
168 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
169
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
170 /// Returns true if this token starts a DeclarationDefinition.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
171 bool isDeclDefStart()
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
172 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
173 return isDeclDefStartToken(kind);
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
174 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
175
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
176 /// Returns true if this token starts a Statement.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
177 bool isStatementStart()
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
178 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
179 return isStatementStartToken(kind);
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
180 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
181
681
95a3c28c0f64 Renamed AsmStatement->AsmBlockStatement and AsnInstruction->AsmStatement.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 679
diff changeset
182 /// Returns true if this token starts an AsmStatement.
95a3c28c0f64 Renamed AsmStatement->AsmBlockStatement and AsnInstruction->AsmStatement.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 679
diff changeset
183 bool isAsmStatementStart()
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
184 {
681
95a3c28c0f64 Renamed AsmStatement->AsmBlockStatement and AsnInstruction->AsmStatement.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 679
diff changeset
185 return isAsmStatementStartToken(kind);
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
186 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
187
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
188 int opEquals(TOK kind2)
163
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null.
aziz
parents: 131
diff changeset
189 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
190 return kind == kind2;
163
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null.
aziz
parents: 131
diff changeset
191 }
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
192
773
6dbbb403fc58 Improved the DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 747
diff changeset
193 int opCmp(Token* rhs)
6dbbb403fc58 Improved the DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 747
diff changeset
194 {
6dbbb403fc58 Improved the DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 747
diff changeset
195 return start < rhs.start;
6dbbb403fc58 Improved the DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 747
diff changeset
196 }
6dbbb403fc58 Improved the DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 747
diff changeset
197
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
198 /// Returns the Location of this token.
675
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
199 Location getLocation(bool realLocation)()
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
200 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
201 auto search_t = this.prev;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
202 // Find previous newline token.
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
203 while (search_t.kind != TOK.Newline)
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
204 search_t = search_t.prev;
675
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
205 static if (realLocation)
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
206 {
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
207 auto filePath = search_t.newline.filePaths.oriPath;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
208 auto lineNum = search_t.newline.oriLineNum;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
209 }
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
210 else
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
211 {
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
212 auto filePath = search_t.newline.filePaths.setPath;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
213 auto lineNum = search_t.newline.oriLineNum - search_t.newline.setLineNum;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
214 }
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
215 auto lineBegin = search_t.end;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
216 // Determine actual line begin and line number.
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
217 while (1)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
218 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
219 search_t = search_t.next;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
220 if (search_t == this)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
221 break;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
222 // Multiline tokens must be rescanned for newlines.
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
223 if (search_t.isMultiline)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
224 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
225 auto p = search_t.start, end = search_t.end;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
226 while (p != end)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
227 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
228 if (scanNewline(p) == '\n')
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
229 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
230 lineBegin = p;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
231 ++lineNum;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
232 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
233 else
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
234 ++p;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
235 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
236 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
237 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
238 return new Location(filePath, lineNum, lineBegin, this.start);
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
239 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
240
675
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
241 alias getLocation!(true) getRealLocation;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
242 alias getLocation!(false) getErrorLocation;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
243
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
244 uint lineCount()
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
245 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
246 uint count = 1;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
247 if (this.isMultiline)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
248 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
249 auto p = this.start, end = this.end;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
250 while (p != end)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
251 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
252 if (scanNewline(p) == '\n')
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
253 ++count;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
254 else
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
255 ++p;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
256 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
257 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
258 return count;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
259 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
260
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
261 /// Return the source text enclosed by the left and right token.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
262 static char[] textSpan(Token* left, Token* right)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
263 {
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
264 assert(left.end <= right.start || left is right );
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
265 return left.start[0 .. right.end - left.start];
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
266 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
267
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
268 /// Uses malloc() to allocate memory for a token.
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
269 new(size_t size)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
270 {
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
271 void* p = malloc(size);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
272 if (p is null)
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
273 throw new OutOfMemoryException(__FILE__, __LINE__);
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
274 *cast(Token*)p = Token.init;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
275 return p;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
276 }
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
277
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
278 /// Deletes a token using free().
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
279 delete(void* p)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
280 {
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
281 auto token = cast(Token*)p;
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
282 if (token)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
283 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
284 if(token.kind == TOK.HashLine)
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
285 token.destructHashLineToken();
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
286 else
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
287 {
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
288 version(D2)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
289 if (token.isTokenStringLiteral)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
290 token.destructTokenStringLiteral();
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
291 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
292 }
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
293 free(p);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
294 }
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
295
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
296 void destructHashLineToken()
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
297 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
298 assert(kind == TOK.HashLine);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
299 delete tokLineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
300 delete tokLineFilespec;
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
301 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
302
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
303 version(D2)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
304 {
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
305 void destructTokenStringLiteral()
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
306 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
307 assert(kind == TOK.String);
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
308 assert(start && *start == 'q' && start[1] == '{');
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
309 assert(tok_str !is null);
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
310 auto tok_it = tok_str;
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
311 auto tok_del = tok_str;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
312 while (tok_it && tok_it.kind != TOK.EOF)
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
313 {
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
314 tok_it = tok_it.next;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
315 assert(tok_del && tok_del.kind != TOK.EOF);
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
316 delete tok_del;
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
317 tok_del = tok_it;
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
318 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
319 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
320 }
208
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
321 }
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
322
681
95a3c28c0f64 Renamed AsmStatement->AsmBlockStatement and AsnInstruction->AsmStatement.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 679
diff changeset
323 /// Data associated with newline tokens.
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
324 struct NewlineData
487
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
325 {
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
326 struct FilePaths
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
327 {
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
328 char[] oriPath; /// Original path to the source text.
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
329 char[] setPath; /// Path set by #line.
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
330 }
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
331 FilePaths* filePaths;
487
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
332 uint oriLineNum; /// Actual line number in the source text.
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
333 uint setLineNum; /// Delta line number set by #line.
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
334 }
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
335
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
336 /// Returns true if this token starts a DeclarationDefinition.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
337 bool isDeclDefStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
338 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
339 switch (tok)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
340 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
341 alias TOK T;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
342 case T.Align, T.Pragma, T.Export, T.Private, T.Package, T.Protected,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
343 T.Public, T.Extern, T.Deprecated, T.Override, T.Abstract,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
344 T.Synchronized, T.Static, T.Final, T.Const, T.Invariant/*D 2.0*/,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
345 T.Auto, T.Scope, T.Alias, T.Typedef, T.Import, T.Enum, T.Class,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
346 T.Interface, T.Struct, T.Union, T.This, T.Tilde, T.Unittest, T.Debug,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
347 T.Version, T.Template, T.New, T.Delete, T.Mixin, T.Semicolon,
508
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
348 T.Identifier, T.Dot, T.Typeof:
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
349 return true;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
350 default:
508
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
351 if (IntegralTypeBegin <= tok && tok <= IntegralTypeEnd)
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
352 return true;
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
353 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
354 return false;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
355 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
356
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
357 /// Returns true if this token starts a Statement.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
358 bool isStatementStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
359 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
360 switch (tok)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
361 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
362 alias TOK T;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
363 case T.Align, T.Extern, T.Final, T.Const, T.Auto, T.Identifier, T.Dot,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
364 T.Typeof, T.If, T.While, T.Do, T.For, T.Foreach, T.Foreach_reverse,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
365 T.Switch, T.Case, T.Default, T.Continue, T.Break, T.Return, T.Goto,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
366 T.With, T.Synchronized, T.Try, T.Throw, T.Scope, T.Volatile, T.Asm,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
367 T.Pragma, T.Mixin, T.Static, T.Debug, T.Version, T.Alias, T.Semicolon,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
368 T.Enum, T.Class, T.Interface, T.Struct, T.Union, T.LBrace, T.Typedef,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
369 T.This, T.Super, T.Null, T.True, T.False, T.Int32, T.Int64, T.Uint32,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
370 T.Uint64, T.Float32, T.Float64, T.Float80, T.Imaginary32,
507
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 502
diff changeset
371 T.Imaginary64, T.Imaginary80, T.CharLiteral, T.String, T.LBracket,
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 502
diff changeset
372 T.Function, T.Delegate, T.Assert, T.Import, T.Typeid, T.Is, T.LParen,
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 502
diff changeset
373 T.Traits/*D2.0*/, T.AndBinary, T.PlusPlus, T.MinusMinus, T.Mul,
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 502
diff changeset
374 T.Minus, T.Plus, T.Not, T.Tilde, T.New, T.Delete, T.Cast:
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
375 return true;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
376 default:
508
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
377 if (IntegralTypeBegin <= tok && tok <= IntegralTypeEnd ||
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
378 SpecialTokensBegin <= tok && tok <= SpecialTokensEnd)
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
379 return true;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
380 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
381 return false;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
382 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
383
681
95a3c28c0f64 Renamed AsmStatement->AsmBlockStatement and AsnInstruction->AsmStatement.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 679
diff changeset
384 /// Returns true if this token starts an AsmStatement.
95a3c28c0f64 Renamed AsmStatement->AsmBlockStatement and AsnInstruction->AsmStatement.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 679
diff changeset
385 bool isAsmStatementStartToken(TOK tok)
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
386 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
387 switch(tok)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
388 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
389 alias TOK T;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
390 case T.In, T.Int, T.Out, T.Identifier, T.Align, T.Semicolon:
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
391 return true;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
392 default:
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
393 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
394 return false;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
395 }