projects/dil: trunk/src/dil/Token.d annotate

annotate trunk/src/dil/Token.d @ 577:9e811db780a6

Moved LexerFuncs.d to package 'lexer'.

author	Aziz K?ksal <aziz.koeksal@gmail.com>
date	Sat, 05 Jan 2008 17:01:15 +0100
parents	3bc7801c207e
children

rev	line source
0 8ba2570de175 Initial import. aziz parents: diff changeset	1 /++
8ba2570de175 Initial import. aziz parents: diff changeset	2 Author: Aziz Köksal
249 32d354584b28 - Upgraded license notices to GPL3. aziz parents: 239 diff changeset	3 License: GPL3
0 8ba2570de175 Initial import. aziz parents: diff changeset	4 +/
326 4a7359b88c11 - Added package 'dil' to module declarations. aziz parents: 325 diff changeset	5 module dil.Token;
577 9e811db780a6 Moved LexerFuncs.d to package 'lexer'. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 552 diff changeset	6
490 47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	7 import dil.Location;
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 500 diff changeset	8 import dil.Identifier;
577 9e811db780a6 Moved LexerFuncs.d to package 'lexer'. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 552 diff changeset	9 import dil.lexer.Funcs;
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	10 import tango.stdc.stdlib : malloc, free;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	11 import tango.core.Exception;
490 47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	12 import common;
0 8ba2570de175 Initial import. aziz parents: diff changeset	13
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 500 diff changeset	14 public import dil.TokensEnum;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	15
500 41b7f9e439bd Moved enum TOK to new module dil.TokenIDs. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	16 /++
41b7f9e439bd Moved enum TOK to new module dil.TokenIDs. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	17 A Token is a sequence of characters formed by the lexical analyzer.
41b7f9e439bd Moved enum TOK to new module dil.TokenIDs. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	18 +/
0 8ba2570de175 Initial import. aziz parents: diff changeset	19 struct Token
8ba2570de175 Initial import. aziz parents: diff changeset	20 {
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	21 enum Flags : ushort
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	22 {
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	23 None,
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	24 Whitespace = 1, /// Tokens with this flag are ignored by the Parser.
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	25 }
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	26
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	27 TOK type; /// The type of the token.
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	28 Flags flags; /// The flags of the token.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	29 /// Pointers to the next and previous tokens (doubly-linked list.)
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	30 Token* next, prev;
131 ce636f3981cc - Removed TOK.Number. aziz parents: 107 diff changeset	31
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	32 char* ws; /// Start of whitespace characters before token. Null if no WS.
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	33 char* start; /// Start of token in source text.
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	34 char* end; /// Points one past the end of token in source text.
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	35
0 8ba2570de175 Initial import. aziz parents: diff changeset	36 union
8ba2570de175 Initial import. aziz parents: diff changeset	37 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	38 /// For newline tokens.
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	39 struct
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	40 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	41 char[] filePath;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	42 uint lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	43 uint lineNum_hline;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 314 diff changeset	44 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	45 /// For #line tokens.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	46 struct
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	47 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	48 Token* tokLineNum; /// #line number
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	49 Token* tokLineFilespec; /// #line number filespec
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	50 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	51 /// For string tokens.
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 314 diff changeset	52 struct
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 314 diff changeset	53 {
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	54 string str;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	55 char pf; /// Postfix 'c', 'w', 'd' or 0 for none.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	56 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	57 Token* tok_str; /// Points to the contents of a token string stored as a
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	58 /// doubly linked list. The last token is always '}' or
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	59 /// EOF in case end of source text is "q{" EOF.
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	60 }
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 500 diff changeset	61 Identifier* ident;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	62 dchar dchar_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	63 long long_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	64 ulong ulong_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	65 int int_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	66 uint uint_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	67 float float_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	68 double double_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	69 real real_;
0 8ba2570de175 Initial import. aziz parents: diff changeset	70 }
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	71
107 722c05bbd5eb - Implemented parseEnumDeclaration() and added class EnumDeclaration. aziz parents: 97 diff changeset	72 alias srcText identifier;
722c05bbd5eb - Implemented parseEnumDeclaration() and added class EnumDeclaration. aziz parents: 97 diff changeset	73
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	74 /// Returns the text of the token.
65 6c21ae79fbb3 - Renamed function Token.span to Token.srcText. aziz parents: 62 diff changeset	75 string srcText()
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	76 {
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	77 assert(start && end);
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	78 return start[0 .. end - start];
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	79 }
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	80
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	81 /// Returns the preceding whitespace of the token.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	82 string wsChars()
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	83 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	84 assert(ws && start);
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	85 return ws[0 .. start - ws];
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	86 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	87
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	88 /// Find next non-whitespace token. Returns 'this' token if the next token is TOK.EOF or null.
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	89 Token* nextNWS()
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	90 out(token)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	91 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	92 assert(token !is null);
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	93 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	94 body
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	95 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	96 auto token = next;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	97 while (token !is null && token.isWhitespace)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	98 token = token.next;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	99 if (token is null \|\| token.type == TOK.EOF)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	100 return this;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	101 return token;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	102 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	103
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	104 /// Find previous non-whitespace token. Returns 'this' token if the previous token is TOK.HEAD or null.
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	105 Token* prevNWS()
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	106 out(token)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	107 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	108 assert(token !is null);
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	109 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	110 body
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	111 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	112 auto token = prev;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	113 while (token !is null && token.isWhitespace)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	114 token = token.prev;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	115 if (token is null \|\| token.type == TOK.HEAD)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	116 return this;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	117 return token;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	118 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	119
210 b7bde6583d3e - Made toString() static. aziz parents: 208 diff changeset	120 static string toString(TOK tok)
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	121 {
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	122 return tokToString[tok];
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	123 }
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	124
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	125 /// Adds Flags.Whitespace to this token's flags.
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	126 void setWhitespaceFlag()
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	127 {
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	128 this.flags \|= Flags.Whitespace;
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	129 }
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	130
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	131 /++
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	132 Returns true if this is a token that can have newlines in it.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	133 These can be block and nested comments and any string literal
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	134 except for escape string literals.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	135 +/
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	136 bool isMultiline()
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	137 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	138 return type == TOK.String && start[0] != '\\' \|\|
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	139 type == TOK.Comment && start[1] != '/';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	140 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	141
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	142 /// Returns true if this is a keyword token.
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	143 bool isKeyword()
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	144 {
163 f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	145 return KeywordsBegin <= type && type <= KeywordsEnd;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	146 }
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	147
508 943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	148 /// Returns true if this is an integral type token.
943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	149 bool isIntegralType()
943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	150 {
943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	151 return IntegralTypeBegin <= type && type <= IntegralTypeEnd;
943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	152 }
943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	153
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	154 /// Returns true if this is a whitespace token.
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	155 bool isWhitespace()
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	156 {
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	157 return !!(flags & Flags.Whitespace);
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	158 }
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	159
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	160 /// Returns true if this is a special token.
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	161 bool isSpecialToken()
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	162 {
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	163 return SpecialTokensBegin <= type && type <= SpecialTokensEnd;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	164 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	165
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	166 version(D2)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	167 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	168 /// Returns true if this is a token string literal.
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	169 bool isTokenStringLiteral()
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	170 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	171 return type == TOK.String && tok_str !is null;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	172 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	173 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	174
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	175 /// Returns true if this token starts a DeclarationDefinition.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	176 bool isDeclDefStart()
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	177 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	178 return isDeclDefStartToken(type);
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	179 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	180
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	181 /// Returns true if this token starts a Statement.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	182 bool isStatementStart()
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	183 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	184 return isStatementStartToken(type);
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	185 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	186
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	187 /// Returns true if this token starts an AsmInstruction.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	188 bool isAsmInstructionStart()
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	189 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	190 return isAsmInstructionStartToken(type);
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	191 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	192
163 f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	193 int opEquals(TOK type2)
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	194 {
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	195 return type == type2;
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	196 }
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	197
490 47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	198 /// Returns the Location of this token.
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	199 Location getLocation()
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	200 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	201 auto search_t = this.prev;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	202 // Find previous newline token.
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	203 while (search_t.type != TOK.Newline)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	204 search_t = search_t.prev;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	205 auto filePath = search_t.filePath;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	206 auto lineNum = search_t.lineNum - search_t.lineNum_hline;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	207 auto lineBegin = search_t.end;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	208 // Determine actual line begin and line number.
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	209 while (1)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	210 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	211 search_t = search_t.next;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	212 if (search_t == this)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	213 break;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	214 // Multiline tokens must be rescanned for newlines.
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	215 if (search_t.isMultiline)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	216 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	217 auto p = search_t.start, end = search_t.end;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	218 while (p != end)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	219 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	220 if (scanNewline(p) == '\n')
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	221 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	222 lineBegin = p;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	223 ++lineNum;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	224 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	225 else
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	226 ++p;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	227 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	228 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	229 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	230 return new Location(filePath, lineNum, lineBegin, this.start);
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	231 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	232
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	233 uint lineCount()
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	234 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	235 uint count = 1;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	236 if (this.isMultiline)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	237 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	238 auto p = this.start, end = this.end;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	239 while (p != end)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	240 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	241 if (scanNewline(p) == '\n')
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	242 ++count;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	243 else
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	244 ++p;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	245 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	246 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	247 return count;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	248 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	249
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	250 /// Return the source text enclosed by the left and right token.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	251 static char[] textSpan(Token* left, Token* right)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	252 {
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	253 assert(left.end <= right.start \|\| left is right );
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	254 return left.start[0 .. right.end - left.start];
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	255 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	256
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	257 new(size_t size)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	258 {
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	259 void* p = malloc(size);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	260 if (p is null)
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	261 throw new OutOfMemoryException(__FILE__, __LINE__);
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	262 cast(Token)p = Token.init;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	263 return p;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	264 }
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	265
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	266 delete(void* p)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	267 {
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	268 auto token = cast(Token*)p;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	269 if (token)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	270 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	271 if(token.type == TOK.HashLine)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	272 token.destructHashLineToken();
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	273 else
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	274 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	275 version(D2)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	276 if (token.isTokenStringLiteral)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	277 token.destructTokenStringLiteral();
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	278 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	279 }
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	280 free(p);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	281 }
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	282
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	283 void destructHashLineToken()
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	284 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	285 assert(type == TOK.HashLine);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	286 delete tokLineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	287 delete tokLineFilespec;
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	288 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	289
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	290 version(D2)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	291 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	292 void destructTokenStringLiteral()
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	293 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	294 assert(type == TOK.String);
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	295 assert(start && *start == 'q' && start[1] == '{');
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	296 assert(tok_str !is null);
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	297 auto tok_it = tok_str;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	298 auto tok_del = tok_str;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	299 while (tok_it && tok_it.type != TOK.EOF)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	300 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	301 tok_it = tok_it.next;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	302 assert(tok_del && tok_del.type != TOK.EOF);
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	303 delete tok_del;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	304 tok_del = tok_it;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	305 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	306 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	307 }
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	308 }
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	309
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	310 /++
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	311 Not used at the moment. Could be useful if more
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	312 info is needed about the location of nodes/tokens.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	313 +/
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	314 struct NewlineInfo
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	315 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	316 char[] oriPath; /// Original path to the source text.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	317 char[] setPath; /// Path set by #line.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	318 uint oriLineNum; /// Actual line number in the source text.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	319 uint setLineNum; /// Delta line number set by #line.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	320 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	321
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	322 /// Returns true if this token starts a DeclarationDefinition.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	323 bool isDeclDefStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	324 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	325 switch (tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	326 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	327 alias TOK T;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	328 case T.Align, T.Pragma, T.Export, T.Private, T.Package, T.Protected,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	329 T.Public, T.Extern, T.Deprecated, T.Override, T.Abstract,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	330 T.Synchronized, T.Static, T.Final, T.Const, T.Invariant/D 2.0/,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	331 T.Auto, T.Scope, T.Alias, T.Typedef, T.Import, T.Enum, T.Class,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	332 T.Interface, T.Struct, T.Union, T.This, T.Tilde, T.Unittest, T.Debug,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	333 T.Version, T.Template, T.New, T.Delete, T.Mixin, T.Semicolon,
508 943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	334 T.Identifier, T.Dot, T.Typeof:
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	335 return true;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	336 default:
508 943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	337 if (IntegralTypeBegin <= tok && tok <= IntegralTypeEnd)
943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	338 return true;
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	339 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	340 return false;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	341 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	342
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	343 /// Returns true if this token starts a Statement.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	344 bool isStatementStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	345 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	346 switch (tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	347 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	348 alias TOK T;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	349 case T.Align, T.Extern, T.Final, T.Const, T.Auto, T.Identifier, T.Dot,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	350 T.Typeof, T.If, T.While, T.Do, T.For, T.Foreach, T.Foreach_reverse,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	351 T.Switch, T.Case, T.Default, T.Continue, T.Break, T.Return, T.Goto,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	352 T.With, T.Synchronized, T.Try, T.Throw, T.Scope, T.Volatile, T.Asm,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	353 T.Pragma, T.Mixin, T.Static, T.Debug, T.Version, T.Alias, T.Semicolon,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	354 T.Enum, T.Class, T.Interface, T.Struct, T.Union, T.LBrace, T.Typedef,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	355 T.This, T.Super, T.Null, T.True, T.False, T.Int32, T.Int64, T.Uint32,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	356 T.Uint64, T.Float32, T.Float64, T.Float80, T.Imaginary32,
507 996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 502 diff changeset	357 T.Imaginary64, T.Imaginary80, T.CharLiteral, T.String, T.LBracket,
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 502 diff changeset	358 T.Function, T.Delegate, T.Assert, T.Import, T.Typeid, T.Is, T.LParen,
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 502 diff changeset	359 T.Traits/D2.0/, T.AndBinary, T.PlusPlus, T.MinusMinus, T.Mul,
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 502 diff changeset	360 T.Minus, T.Plus, T.Not, T.Tilde, T.New, T.Delete, T.Cast:
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	361 return true;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	362 default:
508 943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	363 if (IntegralTypeBegin <= tok && tok <= IntegralTypeEnd \|\|
943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	364 SpecialTokensBegin <= tok && tok <= SpecialTokensEnd)
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	365 return true;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	366 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	367 return false;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	368 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	369
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	370 /// Returns true if this token starts an AsmInstruction.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	371 bool isAsmInstructionStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	372 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	373 switch(tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	374 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	375 alias TOK T;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	376 case T.In, T.Int, T.Out, T.Identifier, T.Align, T.Semicolon:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	377 return true;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	378 default:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	379 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	380 return false;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	381 }

Mercurial > projects > dil

annotate trunk/src/dil/Token.d @ 577:9e811db780a6