projects/dil: trunk/src/dil/Token.d annotate

annotate trunk/src/dil/Token.d @ 414:9c69615a4876

Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Fixed the while statement that skips to the next valid UTF-8 sequence.

author	Aziz K?ksal <aziz.koeksal@gmail.com>
date	Thu, 27 Sep 2007 10:28:08 +0200
parents	4d9ee8e60712
children	ea8c7459f1c4

rev	line source
0 8ba2570de175 Initial import. aziz parents: diff changeset	1 /++
8ba2570de175 Initial import. aziz parents: diff changeset	2 Author: Aziz Köksal
249 32d354584b28 - Upgraded license notices to GPL3. aziz parents: 239 diff changeset	3 License: GPL3
0 8ba2570de175 Initial import. aziz parents: diff changeset	4 +/
326 4a7359b88c11 - Added package 'dil' to module declarations. aziz parents: 325 diff changeset	5 module dil.Token;
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	6 import common;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	7 import tango.stdc.stdlib : malloc, free;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	8 import tango.core.Exception;
0 8ba2570de175 Initial import. aziz parents: diff changeset	9
8ba2570de175 Initial import. aziz parents: diff changeset	10 struct Position
8ba2570de175 Initial import. aziz parents: diff changeset	11 {
8ba2570de175 Initial import. aziz parents: diff changeset	12 size_t loc;
8ba2570de175 Initial import. aziz parents: diff changeset	13 size_t col;
8ba2570de175 Initial import. aziz parents: diff changeset	14 }
8ba2570de175 Initial import. aziz parents: diff changeset	15
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	16 enum TOK : ushort
3 4bbce78bfb1e - Added TOK enum. aziz parents: 0 diff changeset	17 {
95 0eb4c8a5b32b - Added TOK.Invalid. aziz parents: 84 diff changeset	18 Invalid,
0eb4c8a5b32b - Added TOK.Invalid. aziz parents: 84 diff changeset	19
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	20 /// Flag for whitespace tokens that must be ignored in the parsing phase.
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	21 Whitespace = 0x8000,
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	22 Illegal = 1 \| Whitespace,
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	23 Comment = 2 \| Whitespace,
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	24 Shebang = 3 \| Whitespace,
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	25 HashLine = 4 \| Whitespace,
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	26 Filespec = 5 \| Whitespace,
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	27 Empty = 6,
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	28
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	29 Identifier = 7,
9 5d6968cc751e - Parsing string and character literals now (rudimentary implementation.) aziz parents: 4 diff changeset	30 String,
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	31 CharLiteral, WCharLiteral, DCharLiteral,
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 56 diff changeset	32
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	33 // Special tokens
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	34 FILE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	35 LINE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	36 DATE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	37 TIME,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	38 TIMESTAMP,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	39 VENDOR,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	40 VERSION,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	41
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	42 // Number literals
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 56 diff changeset	43 Int32, Int64, Uint32, Uint64,
97 1a83e5070a84 - Added code for parsing IntNumber- and RealNumberExpressions. aziz parents: 95 diff changeset	44 // Floating point number scanner relies on this order. (FloatXY + 3 == ImaginaryXY)
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	45 Float32, Float64, Float80,
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	46 Imaginary32, Imaginary64, Imaginary80,
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 56 diff changeset	47
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	48
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	49 // Brackets
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 18 diff changeset	50 LParen,
d6adfbd7c513 - Added code for parsing braces. aziz parents: 18 diff changeset	51 RParen,
d6adfbd7c513 - Added code for parsing braces. aziz parents: 18 diff changeset	52 LBracket,
d6adfbd7c513 - Added code for parsing braces. aziz parents: 18 diff changeset	53 RBracket,
d6adfbd7c513 - Added code for parsing braces. aziz parents: 18 diff changeset	54 LBrace,
d6adfbd7c513 - Added code for parsing braces. aziz parents: 18 diff changeset	55 RBrace,
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	56
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	57 Dot, Slice, Ellipses,
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	58
131 ce636f3981cc - Removed TOK.Number. aziz parents: 107 diff changeset	59 // Floating point number operators
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	60 Unordered,
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	61 UorE,
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	62 UorG,
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	63 UorGorE,
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	64 UorL,
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	65 UorLorE,
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	66 LorEorG,
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	67 LorG,
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	68
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	69 // Normal operators
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 32 diff changeset	70 Assign, Equal, NotEqual, Not,
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 36 diff changeset	71 LessEqual, Less,
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 36 diff changeset	72 GreaterEqual, Greater,
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	73 LShiftAssign, LShift,
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 36 diff changeset	74 RShiftAssign,RShift,
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 36 diff changeset	75 URShiftAssign, URShift,
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	76 OrAssign, OrLogical, OrBinary,
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	77 AndAssign, AndLogical, AndBinary,
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	78 PlusAssign, PlusPlus, Plus,
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	79 MinusAssign, MinusMinus, Minus,
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	80 DivAssign, Div,
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	81 MulAssign, Mul,
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	82 ModAssign, Mod,
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	83 XorAssign, Xor,
27 43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	84 CatAssign, Catenate,
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	85 Tilde,
71 b3777cca323c - Added Identity and NotIdentity tokens. aziz parents: 65 diff changeset	86 Identity, NotIdentity,
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	87
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	88 Colon,
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	89 Semicolon,
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	90 Question,
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	91 Comma,
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	92 Dollar,
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	93
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	94 /* Keywords:
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	95 NB.: Token.isKeyword() depends on this list being contiguous.
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	96 */
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	97 Abstract,Alias,Align,Asm,Assert,Auto,Body,
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	98 Bool,Break,Byte,Case,Cast,Catch,Cdouble,
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	99 Cent,Cfloat,Char,Class,Const,Continue,Creal,
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	100 Dchar,Debug,Default,Delegate,Delete,Deprecated,Do,
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	101 Double,Else,Enum,Export,Extern,False,Final,
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	102 Finally,Float,For,Foreach,Foreach_reverse,Function,Goto,
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	103 Idouble,If,Ifloat,Import,In,Inout,Int,
269 a416e09c08ea - Implemented D 2.0 additions. aziz parents: 249 diff changeset	104 Interface,Invariant,Ireal,Is,Lazy,Long,Macro/+D2.0+/,
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	105 Mixin,Module,New,Null,Out,Override,Package,
269 a416e09c08ea - Implemented D 2.0 additions. aziz parents: 249 diff changeset	106 Pragma,Private,Protected,Public,Real,Ref/+D2.0+/,Return,
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	107 Scope,Short,Static,Struct,Super,Switch,Synchronized,
269 a416e09c08ea - Implemented D 2.0 additions. aziz parents: 249 diff changeset	108 Template,This,Throw,Traits/+D2.0+/,True,Try,Typedef,Typeid,
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	109 Typeof,Ubyte,Ucent,Uint,Ulong,Union,Unittest,
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	110 Ushort,Version,Void,Volatile,Wchar,While,With,
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	111
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	112 HEAD, // start of linked list
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	113 EOF,
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	114 MAX
3 4bbce78bfb1e - Added TOK enum. aziz parents: 0 diff changeset	115 }
4bbce78bfb1e - Added TOK enum. aziz parents: 0 diff changeset	116
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	117 alias TOK.Abstract KeywordsBegin;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	118 alias TOK.With KeywordsEnd;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	119
0 8ba2570de175 Initial import. aziz parents: diff changeset	120 struct Token
8ba2570de175 Initial import. aziz parents: diff changeset	121 {
3 4bbce78bfb1e - Added TOK enum. aziz parents: 0 diff changeset	122 TOK type;
84 ac8d961d10d1 - Added code for parsing This-,Super-,Null-,Bool-,Dollar-,CharLiteral- and StringLiteralExpression. aziz parents: 82 diff changeset	123 // Position pos;
0 8ba2570de175 Initial import. aziz parents: diff changeset	124
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	125 Token* next, prev;
131 ce636f3981cc - Removed TOK.Number. aziz parents: 107 diff changeset	126
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	127 char* ws; /// Start of whitespace characters before token. Null if no WS.
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	128 char* start; /// Start of token in source text.
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	129 char* end; /// Points one past the end of token in source text.
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	130
0 8ba2570de175 Initial import. aziz parents: diff changeset	131 union
8ba2570de175 Initial import. aziz parents: diff changeset	132 {
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	133 struct
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	134 {
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 314 diff changeset	135 Token* line_num; // #line number
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 314 diff changeset	136 Token* line_filespec; // #line number filespec
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 314 diff changeset	137 }
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 314 diff changeset	138 struct
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 314 diff changeset	139 {
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	140 string str;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	141 char pf; /// Postfix 'c', 'w' or 'd'
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	142 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	143 Token* tok_str; /// Points to the contents of a token string stored as a
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	144 /// doubly linked list. The last token is always '}' or
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 372 diff changeset	145 /// EOF in case end of source text is "q{" EOF.
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	146 }
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	147 dchar dchar_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	148 long long_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	149 ulong ulong_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	150 int int_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	151 uint uint_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	152 float float_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	153 double double_;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 58 diff changeset	154 real real_;
0 8ba2570de175 Initial import. aziz parents: diff changeset	155 }
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	156
107 722c05bbd5eb - Implemented parseEnumDeclaration() and added class EnumDeclaration. aziz parents: 97 diff changeset	157 alias srcText identifier;
722c05bbd5eb - Implemented parseEnumDeclaration() and added class EnumDeclaration. aziz parents: 97 diff changeset	158
65 6c21ae79fbb3 - Renamed function Token.span to Token.srcText. aziz parents: 62 diff changeset	159 string srcText()
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	160 {
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	161 assert(start && end);
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	162 return start[0 .. end - start];
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	163 }
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	164
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	165 /// Find next non-whitespace token. Returns 'this' token if the next token is TOK.EOF or null.
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	166 Token* nextNWS()
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	167 out(token)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	168 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	169 assert(token !is null);
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	170 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	171 body
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	172 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	173 auto token = next;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	174 while (token !is null && token.isWhitespace)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	175 token = token.next;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	176 if (token is null \|\| token.type == TOK.EOF)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	177 return this;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	178 return token;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	179 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	180
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	181 /// Find previous non-whitespace token. Returns 'this' token if the previous token is TOK.HEAD or null.
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	182 Token* prevNWS()
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	183 out(token)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	184 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	185 assert(token !is null);
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	186 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	187 body
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	188 {
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	189 auto token = prev;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	190 while (token !is null && token.isWhitespace)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	191 token = token.prev;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	192 if (token is null \|\| token.type == TOK.HEAD)
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	193 return this;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	194 return token;
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	195 }
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	196
210 b7bde6583d3e - Made toString() static. aziz parents: 208 diff changeset	197 static string toString(TOK tok)
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	198 {
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	199 return tokToString[tok];
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	200 }
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	201
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	202 bool isKeyword()
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	203 {
163 f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	204 return KeywordsBegin <= type && type <= KeywordsEnd;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	205 }
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	206
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	207 bool isWhitespace()
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	208 {
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	209 return !!(type & TOK.Whitespace);
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	210 }
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	211
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	212 bool isSpecialToken()
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	213 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	214 return *start == '_' && type != TOK.Identifier;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	215 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 326 diff changeset	216
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	217 version(D2)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	218 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	219 bool isTokenStringLiteral()
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	220 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	221 return type == TOK.String && tok_str !is null;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	222 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	223 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	224
163 f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	225 int opEquals(TOK type2)
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	226 {
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	227 return type == type2;
f27a98bb17c7 - Fix: when parsing Declarator fails, type and ident is set to null. aziz parents: 131 diff changeset	228 }
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	229
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	230 new(size_t size)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	231 {
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	232 void* p = malloc(size);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	233 if (p is null)
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	234 throw new OutOfMemoryException(__FILE__, __LINE__);
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	235 cast(Token)p = Token.init;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	236 return p;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	237 }
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	238
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	239 delete(void* p)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	240 {
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	241 auto token = cast(Token*)p;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	242 if (token)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	243 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	244 if(token.type == TOK.HashLine)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	245 token.destructHashLineToken();
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	246 else
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	247 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	248 version(D2)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	249 if (token.isTokenStringLiteral)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	250 token.destructTokenStringLiteral();
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	251 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	252 }
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	253 free(p);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 210 diff changeset	254 }
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	255
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	256 void destructHashLineToken()
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	257 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	258 assert(type == TOK.HashLine);
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	259 delete line_num;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	260 delete line_filespec;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	261 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	262
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	263 version(D2)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	264 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	265 void destructTokenStringLiteral()
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	266 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	267 assert(type == TOK.String);
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	268 assert(start && *start == 'q' && start[1] == '{');
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	269 assert(tok_str !is null);
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	270 auto tok_it = tok_str;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	271 auto tok_del = tok_str;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	272 while (tok_it && tok_it.type != TOK.EOF)
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	273 {
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	274 tok_it = tok_it.next;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	275 assert(tok_del && tok_del.type != TOK.EOF);
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	276 delete tok_del;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	277 tok_del = tok_it;
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	278 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	279 }
4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	280 }
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	281 }
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	282
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	283 const string[] tokToString = [
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	284 "Invalid",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	285
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	286 "Illegal",
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	287 "Comment",
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	288 "#! /shebang/",
ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	289 "#line",
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	290 `"filespec"`,
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	291 "Empty",
314 ebd21bbf296e - Added Whitespace, Sheband and Hashline to enum TOK. TOK.Whitespace is a flag and tokens that are considered whitespace are flagged as such. aziz parents: 312 diff changeset	292
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	293 "Identifier",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	294 "String",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	295 "CharLiteral", "WCharLiteral", "DCharLiteral",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	296
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	297 "__FILE__",
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	298 "__LINE__",
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	299 "__DATE__",
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	300 "__TIME__",
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	301 "__TIMESTAMP__",
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	302 "__VENDOR__",
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	303 "__VERSION__",
511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	304
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	305 "Int32", "Int64", "Uint32", "Uint64",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	306 "Float32", "Float64", "Float80",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	307 "Imaginary32", "Imaginary64", "Imaginary80",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	308
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	309 "(",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	310 ")",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	311 "[",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	312 "]",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	313 "{",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	314 "}",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	315
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	316 ".", "..", "...",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	317
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	318 "Unordered",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	319 "UorE",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	320 "UorG",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	321 "UorGorE",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	322 "UorL",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	323 "UorLorE",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	324 "LorEorG",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	325 "LorG",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	326
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	327 "=", "==", "!=", "!",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	328 "<=", "<",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	329 ">=", ">",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	330 "<<=", "<<",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	331 ">>=",">>",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	332 ">>>=", ">>>",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	333 "\|=", "\|\|", "\|",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	334 "&=", "&&", "&",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	335 "+=", "++", "+",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	336 "-=", "--", "-",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	337 "/=", "/",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	338 "=", "",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	339 "%=", "%",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	340 "^=", "^",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	341 "~=", "~",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	342 "~",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	343 "is", "!is",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	344
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	345 ":",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	346 ";",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	347 "?",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	348 ",",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	349 "$",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	350
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	351 "abstract","alias","align","asm","assert","auto","body",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	352 "bool","break","byte","case","cast","catch","cdouble",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	353 "cent","cfloat","char","class","const","continue","creal",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	354 "dchar","debug","default","delegate","delete","deprecated","do",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	355 "double","else","enum","export","extern","false","final",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	356 "finally","float","for","foreach","foreach_reverse","function","goto",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	357 "idouble","if","ifloat","import","in","inout","int",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	358 "interface","invariant","ireal","is","lazy","long","macro",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	359 "mixin","module","new","null","out","override","package",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	360 "pragma","private","protected","public","real","ref","return",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	361 "scope","short","static","struct","super","switch","synchronized",
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	362 "template","this","throw","__traits","true","try","typedef","typeid",
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	363 "typeof","ubyte","ucent","uint","ulong","union","unittest",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	364 "ushort","version","void","volatile","wchar","while","with",
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	365
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	366 "HEAD",
208 0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	367 "EOF"
0a9bccf74046 - Added string table and toString() method to Token. aziz parents: 163 diff changeset	368 ];
359 511c14950cac - Added messages MissingLinkageType and UnrecognizedLinkageType. aziz parents: 343 diff changeset	369 static assert(tokToString.length == TOK.MAX);

Mercurial > projects > dil

annotate trunk/src/dil/Token.d @ 414:9c69615a4876