projects/dil: trunk/src/dil/lexer/Token.d annotate

annotate trunk/src/dil/lexer/Token.d @ 679:ff6971637f88

Renamed Token member type to kind.

author	Aziz K?ksal <aziz.koeksal@gmail.com>
date	Fri, 18 Jan 2008 23:40:12 +0100
parents	e7811328e6c7
children	95a3c28c0f64

rev	line source
0 8ba2570de175 Initial import. aziz parents: diff changeset	1 /++
8ba2570de175 Initial import. aziz parents: diff changeset	2 Author: Aziz Köksal
249 32d354584b28 - Upgraded license notices to GPL3. aziz parents: 239 diff changeset	3 License: GPL3
0 8ba2570de175 Initial import. aziz parents: diff changeset	4 +/
596 39fac5531b85 Moved dil.Token to dil.lexer.Token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 577 diff changeset	5 module dil.lexer.Token;
577 9e811db780a6 Moved LexerFuncs.d to package 'lexer'. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 552 diff changeset	6
600 041eae272362 Moved dil.Identifier to dil.lexer.Identifier. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 597 diff changeset	7 import dil.lexer.Identifier;
041eae272362 Moved dil.Identifier to dil.lexer.Identifier. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 597 diff changeset	8 import dil.lexer.Funcs;
490 47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	9 import dil.Location;
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	10 import tango.stdc.stdlib : malloc, free;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	11 import tango.core.Exception;
490 47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	12 import common;
0 8ba2570de175 Initial import. aziz parents: diff changeset	13
597 4d50267f59c9 Moved dil.TokensEnum to dil.lexer.TokensEnum. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 596 diff changeset	14 public import dil.lexer.TokensEnum;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	15
500 41b7f9e439bd Moved enum TOK to new module dil.TokenIDs. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	16 /++
41b7f9e439bd Moved enum TOK to new module dil.TokenIDs. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	17 A Token is a sequence of characters formed by the lexical analyzer.
41b7f9e439bd Moved enum TOK to new module dil.TokenIDs. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	18 +/
0 8ba2570de175 Initial import. aziz parents: diff changeset	19 struct Token
8ba2570de175 Initial import. aziz parents: diff changeset	20 {
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	21 enum Flags : ushort
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	22 {
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	23 None,
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	24 Whitespace = 1, /// Tokens with this flag are ignored by the Parser.
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	25 }
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	26
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	27 TOK kind; /// The token kind.
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	28 Flags flags; /// The flags of the token.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	29 /// Pointers to the next and previous tokens (doubly-linked list.)
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	30 Token* next, prev;
131 ce636f3981cc - Removed TOK.Number. aziz parents: 107 diff changeset	31
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	32 char* ws; /// Start of whitespace characters before token. Null if no WS.
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	33 char* start; /// Start of token in source text.
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	34 char* end; /// Points one past the end of token in source text.
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	35
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 600 diff changeset	36 // Data associated with this token.
0 8ba2570de175 Initial import. aziz parents: diff changeset	37 union
8ba2570de175 Initial import. aziz parents: diff changeset	38 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	39 /// For newline tokens.
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 600 diff changeset	40 NewlineData newline;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	41 /// For #line tokens.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	42 struct
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	43 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	44 Token* tokLineNum; /// #line number
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	45 Token* tokLineFilespec; /// #line number filespec
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	46 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	47 /// For string tokens.
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 314 diff changeset	48 struct
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 314 diff changeset	49 {
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	50 string str;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	51 char pf; /// Postfix 'c', 'w', 'd' or 0 for none.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	52 version(D2)
612 c65b11c2074c Added getDocComments() to class Node. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	53 Token* tok_str; /++ Points to the contents of a token string stored as a
c65b11c2074c Added getDocComments() to class Node. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	54 doubly linked list. The last token is always '}' or
c65b11c2074c Added getDocComments() to class Node. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	55 EOF in case end of source text is "q{" EOF.
c65b11c2074c Added getDocComments() to class Node. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	56 +/
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	57 }
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 500 diff changeset	58 Identifier* ident;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	59 dchar dchar_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	60 long long_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	61 ulong ulong_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	62 int int_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	63 uint uint_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	64 float float_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	65 double double_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	66 real real_;
0 8ba2570de175 Initial import. aziz parents: diff changeset	67 }
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	68
107 722c05bbd5eb - Implemented parseEnumDeclaration() and added class EnumDeclaration. aziz parents: 97 diff changeset	69 alias srcText identifier;
722c05bbd5eb - Implemented parseEnumDeclaration() and added class EnumDeclaration. aziz parents: 97 diff changeset	70
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	71 /// Returns the text of the token.
65 6c21ae79fbb3 - Renamed function Token.span to Token.srcText. aziz parents: 62 diff changeset	72 string srcText()
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	73 {
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	74 assert(start && end);
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	75 return start[0 .. end - start];
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	76 }
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	77
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	78 /// Returns the preceding whitespace of the token.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	79 string wsChars()
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	80 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	81 assert(ws && start);
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	82 return ws[0 .. start - ws];
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	83 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	84
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	85 /// Find next non-whitespace token. Returns 'this' token if the next token is TOK.EOF or null.
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	86 Token* nextNWS()
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	87 out(token)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	88 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	89 assert(token !is null);
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	90 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	91 body
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	92 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	93 auto token = next;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	94 while (token !is null && token.isWhitespace)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	95 token = token.next;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	96 if (token is null \|\| token.kind == TOK.EOF)
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	97 return this;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	98 return token;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	99 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	100
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	101 /// Find previous non-whitespace token. Returns 'this' token if the previous token is TOK.HEAD or null.
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	102 Token* prevNWS()
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	103 out(token)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	104 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	105 assert(token !is null);
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	106 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	107 body
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	108 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	109 auto token = prev;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	110 while (token !is null && token.isWhitespace)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	111 token = token.prev;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	112 if (token is null \|\| token.kind == TOK.HEAD)
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	113 return this;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	114 return token;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	115 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	116
210 b7bde6583d3e - Made toString() static. aziz parents: 208 diff changeset	117 static string toString(TOK tok)
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	118 {
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	119 return tokToString[tok];
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	120 }
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	121
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	122 /// Adds Flags.Whitespace to this token's flags.
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	123 void setWhitespaceFlag()
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	124 {
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	125 this.flags \|= Flags.Whitespace;
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	126 }
3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	127
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	128 /++
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	129 Returns true if this is a token that can have newlines in it.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	130 These can be block and nested comments and any string literal
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	131 except for escape string literals.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	132 +/
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	133 bool isMultiline()
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	134 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	135 return kind == TOK.String && start[0] != '\\' \|\|
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	136 kind == TOK.Comment && start[1] != '/';
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	137 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	138
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	139 /// Returns true if this is a keyword token.
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	140 bool isKeyword()
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	141 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	142 return KeywordsBegin <= kind && kind <= KeywordsEnd;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	143 }
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	144
508 943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	145 /// Returns true if this is an integral type token.
943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	146 bool isIntegralType()
943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	147 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	148 return IntegralTypeBegin <= kind && kind <= IntegralTypeEnd;
508 943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	149 }
943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	150
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	151 /// Returns true if this is a whitespace token.
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	152 bool isWhitespace()
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	153 {
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 508 diff changeset	154 return !!(flags & Flags.Whitespace);
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	155 }
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	156
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	157 /// Returns true if this is a special token.
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	158 bool isSpecialToken()
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	159 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	160 return SpecialTokensBegin <= kind && kind <= SpecialTokensEnd;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	161 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	162
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	163 version(D2)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	164 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	165 /// Returns true if this is a token string literal.
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	166 bool isTokenStringLiteral()
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	167 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	168 return kind == TOK.String && tok_str !is null;
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	169 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	170 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	171
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	172 /// Returns true if this token starts a DeclarationDefinition.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	173 bool isDeclDefStart()
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	174 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	175 return isDeclDefStartToken(kind);
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	176 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	177
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	178 /// Returns true if this token starts a Statement.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	179 bool isStatementStart()
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	180 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	181 return isStatementStartToken(kind);
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	182 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	183
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	184 /// Returns true if this token starts an AsmInstruction.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	185 bool isAsmInstructionStart()
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	186 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	187 return isAsmInstructionStartToken(kind);
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	188 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	189
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	190 int opEquals(TOK kind2)
163 f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	191 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	192 return kind == kind2;
163 f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	193 }
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	194
490 47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	195 /// Returns the Location of this token.
675 e7811328e6c7 Made Token.getLocation() a template function and added two aliases. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 612 diff changeset	196 Location getLocation(bool realLocation)()
490 47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	197 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	198 auto search_t = this.prev;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	199 // Find previous newline token.
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	200 while (search_t.kind != TOK.Newline)
490 47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	201 search_t = search_t.prev;
675 e7811328e6c7 Made Token.getLocation() a template function and added two aliases. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 612 diff changeset	202 static if (realLocation)
e7811328e6c7 Made Token.getLocation() a template function and added two aliases. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 612 diff changeset	203 {
e7811328e6c7 Made Token.getLocation() a template function and added two aliases. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 612 diff changeset	204 auto filePath = search_t.newline.filePaths.oriPath;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 612 diff changeset	205 auto lineNum = search_t.newline.oriLineNum;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 612 diff changeset	206 }
e7811328e6c7 Made Token.getLocation() a template function and added two aliases. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 612 diff changeset	207 else
e7811328e6c7 Made Token.getLocation() a template function and added two aliases. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 612 diff changeset	208 {
e7811328e6c7 Made Token.getLocation() a template function and added two aliases. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 612 diff changeset	209 auto filePath = search_t.newline.filePaths.setPath;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 612 diff changeset	210 auto lineNum = search_t.newline.oriLineNum - search_t.newline.setLineNum;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 612 diff changeset	211 }
490 47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	212 auto lineBegin = search_t.end;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	213 // Determine actual line begin and line number.
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	214 while (1)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	215 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	216 search_t = search_t.next;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	217 if (search_t == this)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	218 break;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	219 // Multiline tokens must be rescanned for newlines.
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	220 if (search_t.isMultiline)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	221 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	222 auto p = search_t.start, end = search_t.end;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	223 while (p != end)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	224 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	225 if (scanNewline(p) == '\n')
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	226 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	227 lineBegin = p;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	228 ++lineNum;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	229 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	230 else
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	231 ++p;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	232 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	233 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	234 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	235 return new Location(filePath, lineNum, lineBegin, this.start);
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	236 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	237
675 e7811328e6c7 Made Token.getLocation() a template function and added two aliases. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 612 diff changeset	238 alias getLocation!(true) getRealLocation;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 612 diff changeset	239 alias getLocation!(false) getErrorLocation;
e7811328e6c7 Made Token.getLocation() a template function and added two aliases. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 612 diff changeset	240
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	241 uint lineCount()
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	242 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	243 uint count = 1;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	244 if (this.isMultiline)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	245 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	246 auto p = this.start, end = this.end;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	247 while (p != end)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	248 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	249 if (scanNewline(p) == '\n')
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	250 ++count;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	251 else
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	252 ++p;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	253 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	254 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	255 return count;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	256 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	257
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	258 /// Return the source text enclosed by the left and right token.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	259 static char[] textSpan(Token* left, Token* right)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	260 {
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	261 assert(left.end <= right.start \|\| left is right );
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	262 return left.start[0 .. right.end - left.start];
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	263 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	264
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	265 new(size_t size)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	266 {
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	267 void* p = malloc(size);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	268 if (p is null)
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	269 throw new OutOfMemoryException(__FILE__, __LINE__);
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	270 cast(Token)p = Token.init;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	271 return p;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	272 }
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	273
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	274 delete(void* p)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	275 {
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	276 auto token = cast(Token*)p;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	277 if (token)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	278 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	279 if(token.kind == TOK.HashLine)
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	280 token.destructHashLineToken();
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	281 else
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	282 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	283 version(D2)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	284 if (token.isTokenStringLiteral)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	285 token.destructTokenStringLiteral();
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	286 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	287 }
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	288 free(p);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	289 }
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	290
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	291 void destructHashLineToken()
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	292 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	293 assert(kind == TOK.HashLine);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	294 delete tokLineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	295 delete tokLineFilespec;
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	296 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	297
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	298 version(D2)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	299 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	300 void destructTokenStringLiteral()
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	301 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	302 assert(kind == TOK.String);
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	303 assert(start && *start == 'q' && start[1] == '{');
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	304 assert(tok_str !is null);
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	305 auto tok_it = tok_str;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	306 auto tok_del = tok_str;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	307 while (tok_it && tok_it.kind != TOK.EOF)
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	308 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	309 tok_it = tok_it.next;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 675 diff changeset	310 assert(tok_del && tok_del.kind != TOK.EOF);
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	311 delete tok_del;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	312 tok_del = tok_it;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	313 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	314 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	315 }
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	316 }
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	317
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 600 diff changeset	318 /// Stores data about newlines.
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 600 diff changeset	319 struct NewlineData
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	320 {
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 600 diff changeset	321 struct FilePaths
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 600 diff changeset	322 {
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 600 diff changeset	323 char[] oriPath; /// Original path to the source text.
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 600 diff changeset	324 char[] setPath; /// Path set by #line.
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 600 diff changeset	325 }
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 600 diff changeset	326 FilePaths* filePaths;
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	327 uint oriLineNum; /// Actual line number in the source text.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	328 uint setLineNum; /// Delta line number set by #line.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	329 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	330
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	331 /// Returns true if this token starts a DeclarationDefinition.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	332 bool isDeclDefStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	333 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	334 switch (tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	335 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	336 alias TOK T;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	337 case T.Align, T.Pragma, T.Export, T.Private, T.Package, T.Protected,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	338 T.Public, T.Extern, T.Deprecated, T.Override, T.Abstract,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	339 T.Synchronized, T.Static, T.Final, T.Const, T.Invariant/D 2.0/,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	340 T.Auto, T.Scope, T.Alias, T.Typedef, T.Import, T.Enum, T.Class,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	341 T.Interface, T.Struct, T.Union, T.This, T.Tilde, T.Unittest, T.Debug,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	342 T.Version, T.Template, T.New, T.Delete, T.Mixin, T.Semicolon,
508 943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	343 T.Identifier, T.Dot, T.Typeof:
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	344 return true;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	345 default:
508 943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	346 if (IntegralTypeBegin <= tok && tok <= IntegralTypeEnd)
943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	347 return true;
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	348 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	349 return false;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	350 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	351
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	352 /// Returns true if this token starts a Statement.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	353 bool isStatementStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	354 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	355 switch (tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	356 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	357 alias TOK T;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	358 case T.Align, T.Extern, T.Final, T.Const, T.Auto, T.Identifier, T.Dot,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	359 T.Typeof, T.If, T.While, T.Do, T.For, T.Foreach, T.Foreach_reverse,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	360 T.Switch, T.Case, T.Default, T.Continue, T.Break, T.Return, T.Goto,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	361 T.With, T.Synchronized, T.Try, T.Throw, T.Scope, T.Volatile, T.Asm,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	362 T.Pragma, T.Mixin, T.Static, T.Debug, T.Version, T.Alias, T.Semicolon,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	363 T.Enum, T.Class, T.Interface, T.Struct, T.Union, T.LBrace, T.Typedef,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	364 T.This, T.Super, T.Null, T.True, T.False, T.Int32, T.Int64, T.Uint32,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	365 T.Uint64, T.Float32, T.Float64, T.Float80, T.Imaginary32,
507 996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 502 diff changeset	366 T.Imaginary64, T.Imaginary80, T.CharLiteral, T.String, T.LBracket,
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 502 diff changeset	367 T.Function, T.Delegate, T.Assert, T.Import, T.Typeid, T.Is, T.LParen,
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 502 diff changeset	368 T.Traits/D2.0/, T.AndBinary, T.PlusPlus, T.MinusMinus, T.Mul,
996041463028 Removed TOK.WCharLiteral and TOK.DCharLiteral. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 502 diff changeset	369 T.Minus, T.Plus, T.Not, T.Tilde, T.New, T.Delete, T.Cast:
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	370 return true;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	371 default:
508 943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	372 if (IntegralTypeBegin <= tok && tok <= IntegralTypeEnd \|\|
943ecc9c133a Added isIntegralType() to Token and refactored code. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	373 SpecialTokensBegin <= tok && tok <= SpecialTokensEnd)
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	374 return true;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	375 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	376 return false;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	377 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	378
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	379 /// Returns true if this token starts an AsmInstruction.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	380 bool isAsmInstructionStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	381 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	382 switch(tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	383 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	384 alias TOK T;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	385 case T.In, T.Int, T.Out, T.Identifier, T.Align, T.Semicolon:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	386 return true;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	387 default:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	388 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	389 return false;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	390 }

Mercurial > projects > dil

annotate trunk/src/dil/lexer/Token.d @ 679:ff6971637f88