Mercurial > projects > dil

annotate trunk/src/dil/lexer/Token.d @ 798:c24be8d4f6ab

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Added documentation comments.
author Aziz K?ksal <aziz.koeksal@gmail.com>
date Sat, 01 Mar 2008 02:53:06 +0100
parents 8380fb2c765f
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
1 /++
8ba2570de175 Initial import.
aziz
parents:
diff changeset
2 Author: Aziz Köksal
249
32d354584b28 - Upgraded license notices to GPL3.
aziz
parents: 239
diff changeset
3 License: GPL3
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
4 +/
596
39fac5531b85 Moved dil.Token to dil.lexer.Token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
5 module dil.lexer.Token;
577
9e811db780a6 Moved LexerFuncs.d to package 'lexer'.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 552
diff changeset
6
600
041eae272362 Moved dil.Identifier to dil.lexer.Identifier.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 597
diff changeset
7 import dil.lexer.Identifier;
041eae272362 Moved dil.Identifier to dil.lexer.Identifier.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 597
diff changeset
8 import dil.lexer.Funcs;
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
9 import dil.Location;
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
10 import tango.stdc.stdlib : malloc, free;
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
11 import tango.core.Exception;
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
12 import common;
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
13
597
4d50267f59c9 Moved dil.TokensEnum to dil.lexer.TokensEnum.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 596
diff changeset
14 public import dil.lexer.TokensEnum;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 32
diff changeset
15
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
16 /// A Token is a sequence of characters formed by the lexical analyzer.
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
17 struct Token
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
18 { /// Flags set by the Lexer.
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
19 enum Flags : ushort
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
20 {
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
21 None,
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
22 Whitespace = 1, /// Tokens with this flag are ignored by the Parser.
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
23 }
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
24
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
25 TOK kind; /// The token kind.
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
26 Flags flags; /// The flags of the token.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
27 /// Pointers to the next and previous tokens (doubly-linked list.)
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
28 Token* next, prev;
131
ce636f3981cc - Removed TOK.Number.
aziz
parents: 107
diff changeset
29
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
30 /// Start of whitespace characters before token. Null if no WS.
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
31 /// TODO: remove to save space; can be replaced by 'prev.end'.
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
32 char* ws;
798
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
33 char* start; /// Points to the first character of the token.
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
34 char* end; /// Points one character past the end of the token.
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
35
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
36 /// Data associated with this token.
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
37 /// TODO: move data structures out; use only pointers here to keep Token.sizeof small.
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
38 union
8ba2570de175 Initial import.
aziz
parents:
diff changeset
39 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
40 /// For newline tokens.
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
41 NewlineData newline;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
42 /// For #line tokens.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
43 struct
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
44 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
45 Token* tokLineNum; /// #line number
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
46 Token* tokLineFilespec; /// #line number filespec
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
47 }
798
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
48 /// The value of a string token.
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 314
diff changeset
49 struct
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 314
diff changeset
50 {
798
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
51 string str; /// Zero-terminated string. (The zero is included in the length.)
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
52 char pf; /// Postfix 'c', 'w', 'd' or 0 for none.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 372
diff changeset
53 version(D2)
612
c65b11c2074c Added getDocComments() to class Node.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
54 Token* tok_str; /++ Points to the contents of a token string stored as a
c65b11c2074c Added getDocComments() to class Node.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
55 doubly linked list. The last token is always '}' or
c65b11c2074c Added getDocComments() to class Node.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
56 EOF in case end of source text is "q{" EOF.
c65b11c2074c Added getDocComments() to class Node.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
57 +/
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
58 }
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
59 Identifier* ident; /// For keywords and identifiers.
798
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
60 dchar dchar_; /// A character value.
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
61 long long_; /// A long integer value.
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
62 ulong ulong_; /// An unsigned long integer value.
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
63 int int_; /// An integer value.
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
64 uint uint_; /// An unsigned integer value.
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
65 float float_; /// A float value.
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
66 double double_; /// A double value.
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
67 real real_; /// A real value.
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
68 }
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
69
487
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
70 /// Returns the text of the token.
65
6c21ae79fbb3 - Renamed function Token.span to Token.srcText.
aziz
parents: 62
diff changeset
71 string srcText()
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
72 {
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
73 assert(start && end);
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
74 return start[0 .. end - start];
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
75 }
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
76
487
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
77 /// Returns the preceding whitespace of the token.
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
78 string wsChars()
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
79 {
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
80 assert(ws && start);
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
81 return ws[0 .. start - ws];
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
82 }
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
83
798
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
84 /// Finds the next non-whitespace token.
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
85 /// Returns: 'this' token if the next token is TOK.EOF or null.
359
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
86 Token* nextNWS()
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
87 out(token)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
88 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
89 assert(token !is null);
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
90 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
91 body
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
92 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
93 auto token = next;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
94 while (token !is null && token.isWhitespace)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
95 token = token.next;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
96 if (token is null || token.kind == TOK.EOF)
359
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
97 return this;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
98 return token;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
99 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
100
798
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
101 /// Finds the previous non-whitespace token.
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
102 /// Returns: 'this' token if the previous token is TOK.HEAD or null.
359
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
103 Token* prevNWS()
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
104 out(token)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
105 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
106 assert(token !is null);
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
107 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
108 body
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
109 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
110 auto token = prev;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
111 while (token !is null && token.isWhitespace)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
112 token = token.prev;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
113 if (token is null || token.kind == TOK.HEAD)
359
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
114 return this;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
115 return token;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
116 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType.
aziz
parents: 343
diff changeset
117
798
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
118 /// Returns the string for a token kind.
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
119 static string toString(TOK kind)
208
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
120 {
798
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
121 return tokToString[kind];
208
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
122 }
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
123
798
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
124 /// Adds Flags.Whitespace to this.flags.
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
125 void setWhitespaceFlag()
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
126 {
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
127 this.flags |= Flags.Whitespace;
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
128 }
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
129
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
130 /// Returns true if this is a token that can have newlines in it.
798
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
131 ///
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
132 /// These can be block and nested comments and any string literal
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
133 /// except for escape string literals.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
134 bool isMultiline()
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
135 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
136 return kind == TOK.String && start[0] != '\\' ||
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
137 kind == TOK.Comment && start[1] != '/';
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
138 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
139
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
140 /// Returns true if this is a keyword token.
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
141 bool isKeyword()
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
142 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
143 return KeywordsBegin <= kind && kind <= KeywordsEnd;
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
144 }
208
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
145
508
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
146 /// Returns true if this is an integral type token.
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
147 bool isIntegralType()
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
148 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
149 return IntegralTypeBegin <= kind && kind <= IntegralTypeEnd;
508
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
150 }
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
151
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
152 /// Returns true if this is a whitespace token.
314
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such.
aziz
parents: 312
diff changeset
153 bool isWhitespace()
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such.
aziz
parents: 312
diff changeset
154 {
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 508
diff changeset
155 return !!(flags & Flags.Whitespace);
314
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such.
aziz
parents: 312
diff changeset
156 }
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such.
aziz
parents: 312
diff changeset
157
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
158 /// Returns true if this is a special token.
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 326
diff changeset
159 bool isSpecialToken()
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 326
diff changeset
160 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
161 return SpecialTokensBegin <= kind && kind <= SpecialTokensEnd;
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 326
diff changeset
162 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 326
diff changeset
163
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
164 version(D2)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
165 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
166 /// Returns true if this is a token string literal.
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
167 bool isTokenStringLiteral()
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
168 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
169 return kind == TOK.String && tok_str !is null;
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
170 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
171 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
172
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
173 /// Returns true if this token starts a DeclarationDefinition.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
174 bool isDeclDefStart()
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
175 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
176 return isDeclDefStartToken(kind);
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
177 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
178
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
179 /// Returns true if this token starts a Statement.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
180 bool isStatementStart()
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
181 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
182 return isStatementStartToken(kind);
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
183 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
184
681
95a3c28c0f64 Renamed AsmStatement->AsmBlockStatement and AsnInstruction->AsmStatement.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 679
diff changeset
185 /// Returns true if this token starts an AsmStatement.
95a3c28c0f64 Renamed AsmStatement->AsmBlockStatement and AsnInstruction->AsmStatement.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 679
diff changeset
186 bool isAsmStatementStart()
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
187 {
681
95a3c28c0f64 Renamed AsmStatement->AsmBlockStatement and AsnInstruction->AsmStatement.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 679
diff changeset
188 return isAsmStatementStartToken(kind);
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
189 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
190
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
191 int opEquals(TOK kind2)
163
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null.
aziz
parents: 131
diff changeset
192 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
193 return kind == kind2;
163
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null.
aziz
parents: 131
diff changeset
194 }
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
195
773
6dbbb403fc58 Improved the DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 747
diff changeset
196 int opCmp(Token* rhs)
6dbbb403fc58 Improved the DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 747
diff changeset
197 {
6dbbb403fc58 Improved the DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 747
diff changeset
198 return start < rhs.start;
6dbbb403fc58 Improved the DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 747
diff changeset
199 }
6dbbb403fc58 Improved the DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 747
diff changeset
200
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
201 /// Returns the Location of this token.
675
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
202 Location getLocation(bool realLocation)()
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
203 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
204 auto search_t = this.prev;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
205 // Find previous newline token.
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
206 while (search_t.kind != TOK.Newline)
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
207 search_t = search_t.prev;
675
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
208 static if (realLocation)
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
209 {
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
210 auto filePath = search_t.newline.filePaths.oriPath;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
211 auto lineNum = search_t.newline.oriLineNum;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
212 }
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
213 else
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
214 {
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
215 auto filePath = search_t.newline.filePaths.setPath;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
216 auto lineNum = search_t.newline.oriLineNum - search_t.newline.setLineNum;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
217 }
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
218 auto lineBegin = search_t.end;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
219 // Determine actual line begin and line number.
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
220 while (1)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
221 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
222 search_t = search_t.next;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
223 if (search_t == this)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
224 break;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
225 // Multiline tokens must be rescanned for newlines.
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
226 if (search_t.isMultiline)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
227 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
228 auto p = search_t.start, end = search_t.end;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
229 while (p != end)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
230 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
231 if (scanNewline(p) == '\n')
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
232 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
233 lineBegin = p;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
234 ++lineNum;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
235 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
236 else
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
237 ++p;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
238 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
239 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
240 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
241 return new Location(filePath, lineNum, lineBegin, this.start);
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
242 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 487
diff changeset
243
675
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
244 alias getLocation!(true) getRealLocation;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
245 alias getLocation!(false) getErrorLocation;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 612
diff changeset
246
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
247 uint lineCount()
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
248 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
249 uint count = 1;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
250 if (this.isMultiline)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
251 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
252 auto p = this.start, end = this.end;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
253 while (p != end)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
254 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
255 if (scanNewline(p) == '\n')
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
256 ++count;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
257 else
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
258 ++p;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
259 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
260 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
261 return count;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
262 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
263
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
264 /// Return the source text enclosed by the left and right token.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
265 static char[] textSpan(Token* left, Token* right)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
266 {
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
267 assert(left.end <= right.start || left is right );
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
268 return left.start[0 .. right.end - left.start];
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
269 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
270
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
271 /// Uses malloc() to allocate memory for a token.
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
272 new(size_t size)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
273 {
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
274 void* p = malloc(size);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
275 if (p is null)
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
276 throw new OutOfMemoryException(__FILE__, __LINE__);
798
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
277 // TODO: Token.init should be all zeros.
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 783
diff changeset
278 // Maybe use calloc() to avoid this line?
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
279 *cast(Token*)p = Token.init;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
280 return p;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
281 }
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
282
783
8380fb2c765f Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 773
diff changeset
283 /// Deletes a token using free().
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
284 delete(void* p)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
285 {
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
286 auto token = cast(Token*)p;
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
287 if (token)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
288 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
289 if(token.kind == TOK.HashLine)
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
290 token.destructHashLineToken();
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
291 else
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
292 {
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
293 version(D2)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
294 if (token.isTokenStringLiteral)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
295 token.destructTokenStringLiteral();
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
296 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
297 }
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
298 free(p);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 210
diff changeset
299 }
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
300
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
301 void destructHashLineToken()
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
302 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
303 assert(kind == TOK.HashLine);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
304 delete tokLineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 414
diff changeset
305 delete tokLineFilespec;
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
306 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
307
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
308 version(D2)
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
309 {
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
310 void destructTokenStringLiteral()
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
311 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
312 assert(kind == TOK.String);
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
313 assert(start && *start == 'q' && start[1] == '{');
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
314 assert(tok_str !is null);
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
315 auto tok_it = tok_str;
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
316 auto tok_del = tok_str;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
317 while (tok_it && tok_it.kind != TOK.EOF)
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
318 {
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
319 tok_it = tok_it.next;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 675
diff changeset
320 assert(tok_del && tok_del.kind != TOK.EOF);
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
321 delete tok_del;
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
322 tok_del = tok_it;
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
323 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
324 }
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
325 }
208
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
326 }
0a9bccf74046 - Added string table and toString() method to Token.
aziz
parents: 163
diff changeset
327
681
95a3c28c0f64 Renamed AsmStatement->AsmBlockStatement and AsnInstruction->AsmStatement.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 679
diff changeset
328 /// Data associated with newline tokens.
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
329 struct NewlineData
487
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
330 {
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
331 struct FilePaths
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
332 {
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
333 char[] oriPath; /// Original path to the source text.
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
334 char[] setPath; /// Path set by #line.
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
335 }
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
336 FilePaths* filePaths;
487
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
337 uint oriLineNum; /// Actual line number in the source text.
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
338 uint setLineNum; /// Delta line number set by #line.
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
339 }
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
340
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
341 /// Returns true if this token starts a DeclarationDefinition.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
342 bool isDeclDefStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
343 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
344 switch (tok)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
345 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
346 alias TOK T;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
347 case T.Align, T.Pragma, T.Export, T.Private, T.Package, T.Protected,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
348 T.Public, T.Extern, T.Deprecated, T.Override, T.Abstract,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
349 T.Synchronized, T.Static, T.Final, T.Const, T.Invariant/*D 2.0*/,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
350 T.Auto, T.Scope, T.Alias, T.Typedef, T.Import, T.Enum, T.Class,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
351 T.Interface, T.Struct, T.Union, T.This, T.Tilde, T.Unittest, T.Debug,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
352 T.Version, T.Template, T.New, T.Delete, T.Mixin, T.Semicolon,
508
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
353 T.Identifier, T.Dot, T.Typeof:
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
354 return true;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
355 default:
508
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
356 if (IntegralTypeBegin <= tok && tok <= IntegralTypeEnd)
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
357 return true;
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
358 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
359 return false;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
360 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
361
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
362 /// Returns true if this token starts a Statement.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
363 bool isStatementStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
364 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
365 switch (tok)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
366 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
367 alias TOK T;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
368 case T.Align, T.Extern, T.Final, T.Const, T.Auto, T.Identifier, T.Dot,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
369 T.Typeof, T.If, T.While, T.Do, T.For, T.Foreach, T.Foreach_reverse,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
370 T.Switch, T.Case, T.Default, T.Continue, T.Break, T.Return, T.Goto,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
371 T.With, T.Synchronized, T.Try, T.Throw, T.Scope, T.Volatile, T.Asm,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
372 T.Pragma, T.Mixin, T.Static, T.Debug, T.Version, T.Alias, T.Semicolon,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
373 T.Enum, T.Class, T.Interface, T.Struct, T.Union, T.LBrace, T.Typedef,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
374 T.This, T.Super, T.Null, T.True, T.False, T.Int32, T.Int64, T.Uint32,
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
375 T.Uint64, T.Float32, T.Float64, T.Float80, T.Imaginary32,
507
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 502
diff changeset
376 T.Imaginary64, T.Imaginary80, T.CharLiteral, T.String, T.LBracket,
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 502
diff changeset
377 T.Function, T.Delegate, T.Assert, T.Import, T.Typeid, T.Is, T.LParen,
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 502
diff changeset
378 T.Traits/*D2.0*/, T.AndBinary, T.PlusPlus, T.MinusMinus, T.Mul,
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 502
diff changeset
379 T.Minus, T.Plus, T.Not, T.Tilde, T.New, T.Delete, T.Cast:
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
380 return true;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
381 default:
508
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
382 if (IntegralTypeBegin <= tok && tok <= IntegralTypeEnd ||
943ecc9c133a Added isIntegralType() to Token and refactored code.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
383 SpecialTokensBegin <= tok && tok <= SpecialTokensEnd)
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
384 return true;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
385 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
386 return false;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
387 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
388
681
95a3c28c0f64 Renamed AsmStatement->AsmBlockStatement and AsnInstruction->AsmStatement.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 679
diff changeset
389 /// Returns true if this token starts an AsmStatement.
95a3c28c0f64 Renamed AsmStatement->AsmBlockStatement and AsnInstruction->AsmStatement.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 679
diff changeset
390 bool isAsmStatementStartToken(TOK tok)
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
391 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
392 switch(tok)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
393 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
394 alias TOK T;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
395 case T.In, T.Int, T.Out, T.Identifier, T.Align, T.Semicolon:
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
396 return true;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
397 default:
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
398 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
399 return false;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
400 }