projects/dil: trunk/src/dil/Token.d annotate

annotate trunk/src/dil/Token.d @ 497:0ffcc4ff82f3

Refactored a few things in the Lexer. Fix: SpecialTokensEnd should be TOK.VERSION not TOK.Version. Fixed assert statement in textSpan().

author	Aziz K?ksal <aziz.koeksal@gmail.com>
date	Sun, 09 Dec 2007 14:58:38 +0100
parents	5a607597dc22
children	41b7f9e439bd

rev	line source
0 8ba2570de175 Initial import. aziz parents: diff changeset	1 /++
8ba2570de175 Initial import. aziz parents: diff changeset	2 Author: Aziz Köksal
249 32d354584b28 - Upgraded license notices to GPL3. aziz parents: 239 diff changeset	3 License: GPL3
0 8ba2570de175 Initial import. aziz parents: diff changeset	4 +/
326 4a7359b88c11 - Added package 'dil' to module declarations. aziz parents: 325 diff changeset	5 module dil.Token;
490 47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	6 import dil.Location;
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	7 import tango.stdc.stdlib : malloc, free;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	8 import tango.core.Exception;
490 47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	9 import common;
0 8ba2570de175 Initial import. aziz parents: diff changeset	10
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	11 enum TOK : ushort
3 4bbce78bfb1e - Added TOK enum. aziz parents: 0 diff changeset	12 {
95 0eb4c8a5b32b - Added TOK.Invalid. aziz parents: 84 diff changeset	13 Invalid,
0eb4c8a5b32b - Added TOK.Invalid. aziz parents: 84 diff changeset	14
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	15 /// Flag for whitespace tokens that must be ignored in the parsing phase.
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	16 Whitespace = 0x8000,
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	17 Illegal = 1 \| Whitespace,
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	18 Comment = 2 \| Whitespace,
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	19 Shebang = 3 \| Whitespace,
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	20 HashLine = 4 \| Whitespace,
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	21 Filespec = 5 \| Whitespace,
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	22 Newline = 6 \| Whitespace,
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	23 Empty = 7,
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	24
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	25 Identifier = 8,
9 5d6968cc751e - Parsing string and character literals now (rudimentary implementation.) aziz parents: 4 diff changeset	26 String,
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	27 CharLiteral, WCharLiteral, DCharLiteral,
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 56 diff changeset	28
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	29 // Special tokens
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	30 FILE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	31 LINE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	32 DATE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	33 TIME,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	34 TIMESTAMP,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	35 VENDOR,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	36 VERSION,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	37
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	38 // Number literals
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 56 diff changeset	39 Int32, Int64, Uint32, Uint64,
97 1a83e5070a84 - Added code for parsing IntNumber- and RealNumberExpressions. aziz parents: 95 diff changeset	40 // Floating point number scanner relies on this order. (FloatXY + 3 == ImaginaryXY)
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	41 Float32, Float64, Float80,
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	42 Imaginary32, Imaginary64, Imaginary80,
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 56 diff changeset	43
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	44
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	45 // Brackets
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 18 diff changeset	46 LParen,
d6adfbd7c513 - Added code for parsing braces. aziz parents: 18 diff changeset	47 RParen,
d6adfbd7c513 - Added code for parsing braces. aziz parents: 18 diff changeset	48 LBracket,
d6adfbd7c513 - Added code for parsing braces. aziz parents: 18 diff changeset	49 RBracket,
d6adfbd7c513 - Added code for parsing braces. aziz parents: 18 diff changeset	50 LBrace,
d6adfbd7c513 - Added code for parsing braces. aziz parents: 18 diff changeset	51 RBrace,
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	52
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	53 Dot, Slice, Ellipses,
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	54
131 ce636f3981cc - Removed TOK.Number. aziz parents: 107 diff changeset	55 // Floating point number operators
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	56 Unordered,
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	57 UorE,
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	58 UorG,
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	59 UorGorE,
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	60 UorL,
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	61 UorLorE,
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	62 LorEorG,
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	63 LorG,
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	64
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	65 // Normal operators
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	66 Assign, Equal, NotEqual, Not,
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 36 diff changeset	67 LessEqual, Less,
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 36 diff changeset	68 GreaterEqual, Greater,
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	69 LShiftAssign, LShift,
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 36 diff changeset	70 RShiftAssign,RShift,
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 36 diff changeset	71 URShiftAssign, URShift,
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	72 OrAssign, OrLogical, OrBinary,
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	73 AndAssign, AndLogical, AndBinary,
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	74 PlusAssign, PlusPlus, Plus,
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	75 MinusAssign, MinusMinus, Minus,
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	76 DivAssign, Div,
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	77 MulAssign, Mul,
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	78 ModAssign, Mod,
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	79 XorAssign, Xor,
27 43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	80 CatAssign, Catenate,
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	81 Tilde,
71 b3777cca323c - Added Identity and NotIdentity tokens. aziz parents: 65 diff changeset	82 Identity, NotIdentity,
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	83
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	84 Colon,
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	85 Semicolon,
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	86 Question,
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	87 Comma,
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	88 Dollar,
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	89
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	90 /* Keywords:
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	91 NB.: Token.isKeyword() depends on this list being contiguous.
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	92 */
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	93 Abstract,Alias,Align,Asm,Assert,Auto,Body,
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	94 Bool,Break,Byte,Case,Cast,Catch,Cdouble,
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	95 Cent,Cfloat,Char,Class,Const,Continue,Creal,
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	96 Dchar,Debug,Default,Delegate,Delete,Deprecated,Do,
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	97 Double,Else,Enum,Export,Extern,False,Final,
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	98 Finally,Float,For,Foreach,Foreach_reverse,Function,Goto,
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	99 Idouble,If,Ifloat,Import,In,Inout,Int,
269 a416e09c08ea - Implemented D 2.0 additions. aziz parents: 249 diff changeset	100 Interface,Invariant,Ireal,Is,Lazy,Long,Macro/+D2.0+/,
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	101 Mixin,Module,New,Null,Out,Override,Package,
269 a416e09c08ea - Implemented D 2.0 additions. aziz parents: 249 diff changeset	102 Pragma,Private,Protected,Public,Real,Ref/+D2.0+/,Return,
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	103 Scope,Short,Static,Struct,Super,Switch,Synchronized,
269 a416e09c08ea - Implemented D 2.0 additions. aziz parents: 249 diff changeset	104 Template,This,Throw,Traits/+D2.0+/,True,Try,Typedef,Typeid,
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	105 Typeof,Ubyte,Ucent,Uint,Ulong,Union,Unittest,
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	106 Ushort,Version,Void,Volatile,Wchar,While,With,
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	107
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	108 HEAD, // start of linked list
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	109 EOF,
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	110 MAX
3 4bbce78bfb1e - Added TOK enum. aziz parents: 0 diff changeset	111 }
4bbce78bfb1e - Added TOK enum. aziz parents: 0 diff changeset	112
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	113 alias TOK.Abstract KeywordsBegin;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	114 alias TOK.With KeywordsEnd;
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	115 alias TOK.FILE SpecialTokensBegin;
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	116 alias TOK.VERSION SpecialTokensEnd;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	117
0 8ba2570de175 Initial import. aziz parents: diff changeset	118 struct Token
8ba2570de175 Initial import. aziz parents: diff changeset	119 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	120 TOK type; /// The type of the token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	121 /// Pointers to the next and previous tokens (doubly-linked list.)
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	122 Token* next, prev;
131 ce636f3981cc - Removed TOK.Number. aziz parents: 107 diff changeset	123
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	124 char* ws; /// Start of whitespace characters before token. Null if no WS.
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	125 char* start; /// Start of token in source text.
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	126 char* end; /// Points one past the end of token in source text.
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	127
0 8ba2570de175 Initial import. aziz parents: diff changeset	128 union
8ba2570de175 Initial import. aziz parents: diff changeset	129 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	130 /// For newline tokens.
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	131 struct
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	132 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	133 char[] filePath;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	134 uint lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	135 uint lineNum_hline;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 314 diff changeset	136 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	137 /// For #line tokens.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	138 struct
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	139 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	140 Token* tokLineNum; /// #line number
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	141 Token* tokLineFilespec; /// #line number filespec
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	142 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	143 /// For string tokens.
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 314 diff changeset	144 struct
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 314 diff changeset	145 {
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	146 string str;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	147 char pf; /// Postfix 'c', 'w', 'd' or 0 for none.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	148 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	149 Token* tok_str; /// Points to the contents of a token string stored as a
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	150 /// doubly linked list. The last token is always '}' or
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	151 /// EOF in case end of source text is "q{" EOF.
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	152 }
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	153 dchar dchar_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	154 long long_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	155 ulong ulong_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	156 int int_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	157 uint uint_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	158 float float_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	159 double double_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	160 real real_;
0 8ba2570de175 Initial import. aziz parents: diff changeset	161 }
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	162
107 722c05bbd5eb - Implemented parseEnumDeclaration() and added class EnumDeclaration. aziz parents: 97 diff changeset	163 alias srcText identifier;
722c05bbd5eb - Implemented parseEnumDeclaration() and added class EnumDeclaration. aziz parents: 97 diff changeset	164
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	165 /// Returns the text of the token.
65 6c21ae79fbb3 - Renamed function Token.span to Token.srcText. aziz parents: 62 diff changeset	166 string srcText()
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	167 {
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	168 assert(start && end);
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	169 return start[0 .. end - start];
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	170 }
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	171
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	172 /// Returns the preceding whitespace of the token.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	173 string wsChars()
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	174 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	175 assert(ws && start);
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	176 return ws[0 .. start - ws];
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	177 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	178
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	179 /// Find next non-whitespace token. Returns 'this' token if the next token is TOK.EOF or null.
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	180 Token* nextNWS()
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	181 out(token)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	182 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	183 assert(token !is null);
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	184 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	185 body
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	186 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	187 auto token = next;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	188 while (token !is null && token.isWhitespace)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	189 token = token.next;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	190 if (token is null \|\| token.type == TOK.EOF)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	191 return this;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	192 return token;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	193 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	194
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	195 /// Find previous non-whitespace token. Returns 'this' token if the previous token is TOK.HEAD or null.
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	196 Token* prevNWS()
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	197 out(token)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	198 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	199 assert(token !is null);
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	200 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	201 body
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	202 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	203 auto token = prev;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	204 while (token !is null && token.isWhitespace)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	205 token = token.prev;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	206 if (token is null \|\| token.type == TOK.HEAD)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	207 return this;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	208 return token;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	209 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	210
210 b7bde6583d3e - Made toString() static. aziz parents: 208 diff changeset	211 static string toString(TOK tok)
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	212 {
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	213 return tokToString[tok];
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	214 }
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	215
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	216 /++
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	217 Returns true if this is a token that can have newlines in it.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	218 These can be block and nested comments and any string literal
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	219 except for escape string literals.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	220 +/
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	221 bool isMultiline()
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	222 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	223 return type == TOK.String && start[0] != '\\' \|\|
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	224 type == TOK.Comment && start[1] != '/';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	225 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	226
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	227 /// Returns true if this is a keyword token.
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	228 bool isKeyword()
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	229 {
163 f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	230 return KeywordsBegin <= type && type <= KeywordsEnd;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	231 }
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	232
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	233 /// Returns true if this is a whitespace token.
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	234 bool isWhitespace()
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	235 {
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	236 return !!(type & TOK.Whitespace);
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	237 }
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	238
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	239 /// Returns true if this is a special token.
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	240 bool isSpecialToken()
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	241 {
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	242 return SpecialTokensBegin <= type && type <= SpecialTokensEnd;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	243 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	244
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	245 version(D2)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	246 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	247 /// Returns true if this is a token string literal.
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	248 bool isTokenStringLiteral()
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	249 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	250 return type == TOK.String && tok_str !is null;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	251 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	252 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	253
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	254 /// Returns true if this token starts a DeclarationDefinition.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	255 bool isDeclDefStart()
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	256 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	257 return isDeclDefStartToken(type);
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	258 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	259
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	260 /// Returns true if this token starts a Statement.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	261 bool isStatementStart()
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	262 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	263 return isStatementStartToken(type);
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	264 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	265
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	266 /// Returns true if this token starts an AsmInstruction.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	267 bool isAsmInstructionStart()
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	268 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	269 return isAsmInstructionStartToken(type);
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	270 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	271
163 f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	272 int opEquals(TOK type2)
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	273 {
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	274 return type == type2;
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	275 }
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	276
490 47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	277 import dil.LexerFuncs;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	278 /// Returns the Location of this token.
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	279 Location getLocation()
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	280 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	281 auto search_t = this.prev;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	282 // Find previous newline token.
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	283 while (search_t.type != TOK.Newline)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	284 search_t = search_t.prev;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	285 auto filePath = search_t.filePath;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	286 auto lineNum = search_t.lineNum - search_t.lineNum_hline;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	287 auto lineBegin = search_t.end;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	288 // Determine actual line begin and line number.
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	289 while (1)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	290 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	291 search_t = search_t.next;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	292 if (search_t == this)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	293 break;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	294 // Multiline tokens must be rescanned for newlines.
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	295 if (search_t.isMultiline)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	296 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	297 auto p = search_t.start, end = search_t.end;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	298 while (p != end)
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	299 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	300 if (scanNewline(p) == '\n')
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	301 {
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	302 lineBegin = p;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	303 ++lineNum;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	304 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	305 else
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	306 ++p;
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	307 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	308 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	309 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	310 return new Location(filePath, lineNum, lineBegin, this.start);
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	311 }
47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	312
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	313 uint lineCount()
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	314 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	315 uint count = 1;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	316 if (this.isMultiline)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	317 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	318 auto p = this.start, end = this.end;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	319 while (p != end)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	320 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	321 if (scanNewline(p) == '\n')
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	322 ++count;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	323 else
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	324 ++p;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	325 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	326 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	327 return count;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	328 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	329
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	330 /// Return the source text enclosed by the left and right token.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	331 static char[] textSpan(Token* left, Token* right)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	332 {
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	333 assert(left.end <= right.start \|\| left is right );
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	334 return left.start[0 .. right.end - left.start];
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	335 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	336
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	337 new(size_t size)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	338 {
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	339 void* p = malloc(size);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	340 if (p is null)
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	341 throw new OutOfMemoryException(__FILE__, __LINE__);
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	342 cast(Token)p = Token.init;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	343 return p;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	344 }
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	345
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	346 delete(void* p)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	347 {
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	348 auto token = cast(Token*)p;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	349 if (token)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	350 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	351 if(token.type == TOK.HashLine)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	352 token.destructHashLineToken();
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	353 else
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	354 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	355 version(D2)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	356 if (token.isTokenStringLiteral)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	357 token.destructTokenStringLiteral();
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	358 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	359 }
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	360 free(p);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	361 }
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	362
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	363 void destructHashLineToken()
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	364 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	365 assert(type == TOK.HashLine);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	366 delete tokLineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	367 delete tokLineFilespec;
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	368 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	369
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	370 version(D2)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	371 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	372 void destructTokenStringLiteral()
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	373 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	374 assert(type == TOK.String);
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	375 assert(start && *start == 'q' && start[1] == '{');
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	376 assert(tok_str !is null);
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	377 auto tok_it = tok_str;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	378 auto tok_del = tok_str;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	379 while (tok_it && tok_it.type != TOK.EOF)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	380 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	381 tok_it = tok_it.next;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	382 assert(tok_del && tok_del.type != TOK.EOF);
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	383 delete tok_del;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	384 tok_del = tok_it;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	385 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	386 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	387 }
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	388 }
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	389
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	390 /++
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	391 Not used at the moment. Could be useful if more
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	392 info is needed about the location of nodes/tokens.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	393 +/
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	394 struct NewlineInfo
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	395 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	396 char[] oriPath; /// Original path to the source text.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	397 char[] setPath; /// Path set by #line.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	398 uint oriLineNum; /// Actual line number in the source text.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	399 uint setLineNum; /// Delta line number set by #line.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	400 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	401
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	402 /// A table mapping each TOK to a string.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	403 private const string[] tokToString = [
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	404 "Invalid",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	405
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	406 "Illegal",
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	407 "Comment",
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	408 "#! /shebang/",
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	409 "#line",
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	410 `"filespec"`,
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	411 "Newline",
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	412 "Empty",
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	413
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	414 "Identifier",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	415 "String",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	416 "CharLiteral", "WCharLiteral", "DCharLiteral",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	417
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	418 "__FILE__",
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	419 "__LINE__",
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	420 "__DATE__",
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	421 "__TIME__",
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	422 "__TIMESTAMP__",
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	423 "__VENDOR__",
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	424 "__VERSION__",
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	425
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	426 "Int32", "Int64", "Uint32", "Uint64",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	427 "Float32", "Float64", "Float80",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	428 "Imaginary32", "Imaginary64", "Imaginary80",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	429
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	430 "(",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	431 ")",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	432 "[",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	433 "]",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	434 "{",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	435 "}",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	436
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	437 ".", "..", "...",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	438
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	439 "!<>=", // Unordered
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	440 "!<>", // UorE
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	441 "!<=", // UorG
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	442 "!<", // UorGorE
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	443 "!>=", // UorL
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	444 "!>", // UorLorE
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	445 "<>=", // LorEorG
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	446 "<>", // LorG
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	447
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	448 "=", "==", "!=", "!",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	449 "<=", "<",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	450 ">=", ">",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	451 "<<=", "<<",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	452 ">>=",">>",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	453 ">>>=", ">>>",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	454 "\|=", "\|\|", "\|",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	455 "&=", "&&", "&",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	456 "+=", "++", "+",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	457 "-=", "--", "-",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	458 "/=", "/",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	459 "=", "",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	460 "%=", "%",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	461 "^=", "^",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	462 "~=", "~",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	463 "~",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	464 "is", "!is",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	465
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	466 ":",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	467 ";",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	468 "?",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	469 ",",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	470 "$",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	471
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	472 "abstract","alias","align","asm","assert","auto","body",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	473 "bool","break","byte","case","cast","catch","cdouble",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	474 "cent","cfloat","char","class","const","continue","creal",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	475 "dchar","debug","default","delegate","delete","deprecated","do",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	476 "double","else","enum","export","extern","false","final",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	477 "finally","float","for","foreach","foreach_reverse","function","goto",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	478 "idouble","if","ifloat","import","in","inout","int",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	479 "interface","invariant","ireal","is","lazy","long","macro",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	480 "mixin","module","new","null","out","override","package",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	481 "pragma","private","protected","public","real","ref","return",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	482 "scope","short","static","struct","super","switch","synchronized",
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	483 "template","this","throw","__traits","true","try","typedef","typeid",
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	484 "typeof","ubyte","ucent","uint","ulong","union","unittest",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	485 "ushort","version","void","volatile","wchar","while","with",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	486
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	487 "HEAD",
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	488 "EOF"
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	489 ];
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 414 diff changeset	490 static assert(tokToString.length == TOK.MAX);
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	491
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	492 /// Returns true if this token starts a DeclarationDefinition.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	493 bool isDeclDefStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	494 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	495 switch (tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	496 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	497 alias TOK T;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	498 case T.Align, T.Pragma, T.Export, T.Private, T.Package, T.Protected,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	499 T.Public, T.Extern, T.Deprecated, T.Override, T.Abstract,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	500 T.Synchronized, T.Static, T.Final, T.Const, T.Invariant/D 2.0/,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	501 T.Auto, T.Scope, T.Alias, T.Typedef, T.Import, T.Enum, T.Class,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	502 T.Interface, T.Struct, T.Union, T.This, T.Tilde, T.Unittest, T.Debug,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	503 T.Version, T.Template, T.New, T.Delete, T.Mixin, T.Semicolon,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	504 T.Identifier, T.Dot, T.Typeof,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	505 T.Char, T.Wchar, T.Dchar, T.Bool,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	506 T.Byte, T.Ubyte, T.Short, T.Ushort,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	507 T.Int, T.Uint, T.Long, T.Ulong,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	508 T.Float, T.Double, T.Real,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	509 T.Ifloat, T.Idouble, T.Ireal,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	510 T.Cfloat, T.Cdouble, T.Creal, T.Void:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	511 return true;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	512 default:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	513 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	514 return false;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	515 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	516
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	517 /// Returns true if this token starts a Statement.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	518 bool isStatementStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	519 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	520 switch (tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	521 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	522 alias TOK T;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	523 case T.Align, T.Extern, T.Final, T.Const, T.Auto, T.Identifier, T.Dot,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	524 T.Typeof, T.If, T.While, T.Do, T.For, T.Foreach, T.Foreach_reverse,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	525 T.Switch, T.Case, T.Default, T.Continue, T.Break, T.Return, T.Goto,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	526 T.With, T.Synchronized, T.Try, T.Throw, T.Scope, T.Volatile, T.Asm,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	527 T.Pragma, T.Mixin, T.Static, T.Debug, T.Version, T.Alias, T.Semicolon,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	528 T.Enum, T.Class, T.Interface, T.Struct, T.Union, T.LBrace, T.Typedef,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	529 T.This, T.Super, T.Null, T.True, T.False, T.Int32, T.Int64, T.Uint32,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	530 T.Uint64, T.Float32, T.Float64, T.Float80, T.Imaginary32,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	531 T.Imaginary64, T.Imaginary80, T.CharLiteral, T.WCharLiteral,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	532 T.DCharLiteral, T.String, T.LBracket, T.Function, T.Delegate,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	533 T.Assert, T.Import, T.Typeid, T.Is, T.LParen, T.Traits/D2.0/,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	534 T.AndBinary, T.PlusPlus, T.MinusMinus, T.Mul,T.Minus, T.Plus, T.Not,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	535 T.Tilde, T.New, T.Delete, T.Cast:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	536 case T.Char, T.Wchar, T.Dchar, T.Bool,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	537 T.Byte, T.Ubyte, T.Short, T.Ushort,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	538 T.Int, T.Uint, T.Long, T.Ulong,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	539 T.Float, T.Double, T.Real,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	540 T.Ifloat, T.Idouble, T.Ireal,
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	541 T.Cfloat, T.Cdouble, T.Creal, T.Void:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	542 return true;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	543 default:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	544 if (SpecialTokensBegin <= tok && tok <= SpecialTokensEnd)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	545 return true;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	546 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	547 return false;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	548 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	549
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	550 /// Returns true if this token starts an AsmInstruction.
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	551 bool isAsmInstructionStartToken(TOK tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	552 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	553 switch(tok)
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	554 {
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	555 alias TOK T;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	556 case T.In, T.Int, T.Out, T.Identifier, T.Align, T.Semicolon:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	557 return true;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	558 default:
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	559 }
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	560 return false;
5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	561 }

Mercurial > projects > dil

annotate trunk/src/dil/Token.d @ 497:0ffcc4ff82f3