projects/dang: lexer/Token.d annotate

annotate lexer/Token.d @ 126:c3b24e7e8cf8

Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).

author	Anders Johnsen <skabet@gmail.com>
date	Tue, 27 May 2008 10:32:31 +0200
parents	6a5f745d351c
children	2be29b296081

rev	line source
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	1 module lexer.Token;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	2
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	3 public
88 eb5b2c719a39 Major change to locations, tokens and expressions. Anders Halager <halager@gmail.com> parents: 74 diff changeset	4 import basic.SourceLocation;
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	5
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	6 import Integer = tango.text.convert.Integer;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	7
42 4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	8 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	9 The Token struct will be used through the Lexer, Parser and other
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	10 modules as a location into source.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	11
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	12 The Token should always be optimized for size to limit unnecessary
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	13 memory usage.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	14 */
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	15 struct Token
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	16 {
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	17 Tok type;
88 eb5b2c719a39 Major change to locations, tokens and expressions. Anders Halager <halager@gmail.com> parents: 74 diff changeset	18 SLoc location;
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	19 uint length;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	20
42 4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	21 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	22 Create a new token with a Tok type, Location in source and a
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	23 length of how many chars the Token span in the source
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	24 */
88 eb5b2c719a39 Major change to locations, tokens and expressions. Anders Halager <halager@gmail.com> parents: 74 diff changeset	25 static Token opCall (Tok type, SLoc location, uint length)
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	26 {
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	27 Token t;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	28 t.type = type;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	29 t.location = location;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	30 t.length = length;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	31 return t;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	32 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	33
42 4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	34 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	35 Get the type of the Token as a string
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	36 */
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	37 char[] getType ()
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	38 {
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	39 return typeToString[this.type];
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	40 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	41
42 4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	42 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	43 A human readable dump of a Token
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	44 */
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	45 char[] toString ()
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	46 {
88 eb5b2c719a39 Major change to locations, tokens and expressions. Anders Halager <halager@gmail.com> parents: 74 diff changeset	47 return this.getType()~": Len: "~Integer.toString(this.length);
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	48 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	49
88 eb5b2c719a39 Major change to locations, tokens and expressions. Anders Halager <halager@gmail.com> parents: 74 diff changeset	50 /// Get the range of this token
eb5b2c719a39 Major change to locations, tokens and expressions. Anders Halager <halager@gmail.com> parents: 74 diff changeset	51 SourceRange asRange() { return SourceRange(location, location + length); }
44 495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	52
495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	53 /**
495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	54 Returns true if the type of this token is a basic type (int, float, ...).
495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	55 Void is included, although a void in it self is not really a type.
495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	56 */
495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	57 bool isBasicType()
495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	58 {
495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	59 return type >= Tok.Byte && type <= Tok.Void;
495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	60 }
495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	61
495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	62 /**
48 b6c1dc30ca4b Only tests that dont pass now are structs and switches Anders Halager <halager@gmail.com> parents: 44 diff changeset	63 Returns true for all the various assignments (=, +=, *= ...)
b6c1dc30ca4b Only tests that dont pass now are structs and switches Anders Halager <halager@gmail.com> parents: 44 diff changeset	64 */
b6c1dc30ca4b Only tests that dont pass now are structs and switches Anders Halager <halager@gmail.com> parents: 44 diff changeset	65 bool isAssignment()
b6c1dc30ca4b Only tests that dont pass now are structs and switches Anders Halager <halager@gmail.com> parents: 44 diff changeset	66 {
126 c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	67 return type >= Tok.Assign && type <= Tok.PercentAssign;
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	68 }
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	69
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	70 /**
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	71 Returns true for all attributes( public, static, private...)
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	72 */
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	73 bool isAttribute()
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	74 {
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	75 return type >= Tok.Public && type <= Tok.Auto;
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	76 }
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	77
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	78 /**
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	79 just a shortcut to avoid `token.type == tok.Switch`.
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	80 */
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	81 bool isSwitch()
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	82 {
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	83 return type == Tok.Switch;
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	84 }
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	85
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	86 /**
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	87 just a shortcut to avoid `token.type == tok.While`.
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	88 */
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	89 bool isWhile()
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	90 {
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	91 return type == Tok.While;
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	92 }
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	93
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	94 /**
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	95 just a shortcut to avoid `token.type == tok.If`.
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	96 */
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	97 bool isIf()
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	98 {
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	99 return type == Tok.If;
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	100 }
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	101
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	102 /**
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	103 just a shortcut to avoid `token.type == tok.Return`.
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	104 */
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	105 bool isReturn()
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	106 {
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	107 return type == Tok.Return;
48 b6c1dc30ca4b Only tests that dont pass now are structs and switches Anders Halager <halager@gmail.com> parents: 44 diff changeset	108 }
b6c1dc30ca4b Only tests that dont pass now are structs and switches Anders Halager <halager@gmail.com> parents: 44 diff changeset	109
b6c1dc30ca4b Only tests that dont pass now are structs and switches Anders Halager <halager@gmail.com> parents: 44 diff changeset	110 /**
44 495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	111 Just a shortcut to avoid `token.type == Tok.Identifier`.
495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	112 */
495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	113 bool isIdentifier()
495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	114 {
495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	115 return type == Tok.Identifier;
495188f9078e Big update - Moving towards a better, more seperated parser Anders Halager <halager@gmail.com> parents: 42 diff changeset	116 }
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	117 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	118
42 4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	119 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	120 Tok is short for TokenType. This enum list is to supply the Token
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	121 with a type.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	122
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	123 This enum is used to switch over "many" places.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	124 */
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	125 enum Tok : ushort
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	126 {
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	127 /* Non-code related tokens */
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	128 EOF,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	129
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	130 /* Basic types */
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	131 Identifier,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	132 Integer,
126 c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	133 String,
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	134
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	135 /* Basic operators */
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	136 Assign,
126 c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	137 PlusAssign,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	138 MinusAssign,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	139 StarAssign,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	140 SlashAssign,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	141 PercentAssign,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	142 Plus,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	143 Minus,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	144 Star,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	145 Slash,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	146 Percent,
123 6a5f745d351c Parsing <<, >> and >>>. Anders Johnsen <skabet@gmail.com> parents: 104 diff changeset	147 LeftShift, RightShift, UnsignedRightShift,
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	148 Comma,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	149
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	150 /* Symbols */
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	151 OpenParentheses,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	152 CloseParentheses,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	153 OpenBrace,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	154 CloseBrace,
72 628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 68 diff changeset	155 OpenBracket,
628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 68 diff changeset	156 CloseBracket,
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	157 Seperator,
36 ce17bea8e9bd Switch statements support Anders Halager <halager@gmail.com> parents: 28 diff changeset	158 Colon,
28 69464d465284 Now supporting structs - both read and write. Still a few errors though, so watch out. Anders Johnsen <skabet@gmail.com> parents: 22 diff changeset	159 Dot,
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	160
8 2e1069ee21af Added Ne, Lt, Le, Gt, Ge and Not in lexer johnsen@johnsen-desktop parents: 6 diff changeset	161 /* Comparator operators */
2e1069ee21af Added Ne, Lt, Le, Gt, Ge and Not in lexer johnsen@johnsen-desktop parents: 6 diff changeset	162 Eq, Ne,
2e1069ee21af Added Ne, Lt, Le, Gt, Ge and Not in lexer johnsen@johnsen-desktop parents: 6 diff changeset	163 Lt, Gt,
2e1069ee21af Added Ne, Lt, Le, Gt, Ge and Not in lexer johnsen@johnsen-desktop parents: 6 diff changeset	164 Le, Ge,
2e1069ee21af Added Ne, Lt, Le, Gt, Ge and Not in lexer johnsen@johnsen-desktop parents: 6 diff changeset	165
2e1069ee21af Added Ne, Lt, Le, Gt, Ge and Not in lexer johnsen@johnsen-desktop parents: 6 diff changeset	166 Not,
6 606a57c90a0b Now lexing == as Equals johnsen@johnsen-desktop parents: 5 diff changeset	167
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	168 /* Keywords */
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	169 Byte, Ubyte,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	170 Short, Ushort,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	171 Int, Uint,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	172 Long, Ulong,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	173
103 09b4d74cb3f5 Parsing char, wchar and dchar as basic types. Anders Johnsen <skabet@gmail.com> parents: 94 diff changeset	174 Char, Wchar, Dchar,
09b4d74cb3f5 Parsing char, wchar and dchar as basic types. Anders Johnsen <skabet@gmail.com> parents: 94 diff changeset	175
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	176 Float, Double,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	177
5 2c5a8f4c254a Added very simple if support. Anders Halager <halager@gmail.com> parents: 1 diff changeset	178 Bool,
2c5a8f4c254a Added very simple if support. Anders Halager <halager@gmail.com> parents: 1 diff changeset	179
37 858b9805843d Bug-fixes Anders Halager <halager@gmail.com> parents: 36 diff changeset	180 Void,
858b9805843d Bug-fixes Anders Halager <halager@gmail.com> parents: 36 diff changeset	181
126 c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	182 Struct, Function, Delegate, Class,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	183 Interface, Union, Typedef, Typeid,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	184 Typeof, Sizeof, Alias,
22 e331e4e816e4 now handling structs to some extend johnsen@johnsen-laptop parents: 21 diff changeset	185
11 642c6a998fd9 Support for while statements and fixed scope for if Anders Halager <halager@gmail.com> parents: 8 diff changeset	186 If, Else,
642c6a998fd9 Support for while statements and fixed scope for if Anders Halager <halager@gmail.com> parents: 8 diff changeset	187 While,
126 c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	188 Switch, Case, Default, Break,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	189 Return, Cast,
104 7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST. Anders Johnsen <skabet@gmail.com> parents: 103 diff changeset	190
94 48bb2287c035 Added Modules. Right now it's very simple - will grow with time and need. Anders Johnsen <skabet@gmail.com> parents: 88 diff changeset	191 Module, Import,
48bb2287c035 Added Modules. Right now it's very simple - will grow with time and need. Anders Johnsen <skabet@gmail.com> parents: 88 diff changeset	192
126 c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	193 /* Attributes */
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	194 Public, Private, Package, Protected, Export,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	195 Static,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	196 Final,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	197 Const,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	198 Abstract,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	199 Override,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	200 Depracted,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	201 Auto,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	202
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	203 Align,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	204
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	205 Asm,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	206
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	207 In, Out, Body,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	208
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	209 Assert, Throw, Try, Catch, Finally,
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	210
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	211
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	212
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	213
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	214 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	215
42 4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	216 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	217 An associative array to supply a Tok to String function.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	218
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	219 Keep always this list updated when adding a new Tok.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 37 diff changeset	220 */
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	221 public char[][Tok] typeToString;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	222
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	223 static this()
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	224 {
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	225 typeToString =
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	226 [
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	227 Tok.EOF:"EOF"[],
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	228 Tok.Identifier:"Identifier",
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	229 Tok.Byte:"Byte",
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	230 Tok.Short:"Short",
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	231 Tok.Int:"Int",
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	232 Tok.Long:"Long",
103 09b4d74cb3f5 Parsing char, wchar and dchar as basic types. Anders Johnsen <skabet@gmail.com> parents: 94 diff changeset	233 Tok.Char:"Char",
09b4d74cb3f5 Parsing char, wchar and dchar as basic types. Anders Johnsen <skabet@gmail.com> parents: 94 diff changeset	234 Tok.Wchar:"Wchar",
09b4d74cb3f5 Parsing char, wchar and dchar as basic types. Anders Johnsen <skabet@gmail.com> parents: 94 diff changeset	235 Tok.Dchar:"Dchar",
5 2c5a8f4c254a Added very simple if support. Anders Halager <halager@gmail.com> parents: 1 diff changeset	236 Tok.Bool:"Bool",
37 858b9805843d Bug-fixes Anders Halager <halager@gmail.com> parents: 36 diff changeset	237 Tok.Void:"Void",
8 2e1069ee21af Added Ne, Lt, Le, Gt, Ge and Not in lexer johnsen@johnsen-desktop parents: 6 diff changeset	238 Tok.Eq:"Eq",
2e1069ee21af Added Ne, Lt, Le, Gt, Ge and Not in lexer johnsen@johnsen-desktop parents: 6 diff changeset	239 Tok.Ne:"Ne",
2e1069ee21af Added Ne, Lt, Le, Gt, Ge and Not in lexer johnsen@johnsen-desktop parents: 6 diff changeset	240 Tok.Lt:"Lt",
2e1069ee21af Added Ne, Lt, Le, Gt, Ge and Not in lexer johnsen@johnsen-desktop parents: 6 diff changeset	241 Tok.Le:"Le",
2e1069ee21af Added Ne, Lt, Le, Gt, Ge and Not in lexer johnsen@johnsen-desktop parents: 6 diff changeset	242 Tok.Gt:"Gt",
2e1069ee21af Added Ne, Lt, Le, Gt, Ge and Not in lexer johnsen@johnsen-desktop parents: 6 diff changeset	243 Tok.Ge:"Ge",
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	244 Tok.OpenParentheses:"OpenParentheses",
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	245 Tok.CloseParentheses:"CloseParentheses",
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	246 Tok.OpenBrace:"OpenBrace",
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	247 Tok.CloseBrace:"CloseBrace",
72 628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 68 diff changeset	248 Tok.OpenBracket:"OpenBracket",
628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 68 diff changeset	249 Tok.CloseBracket:"CloseBracket",
28 69464d465284 Now supporting structs - both read and write. Still a few errors though, so watch out. Anders Johnsen <skabet@gmail.com> parents: 22 diff changeset	250 Tok.Dot:"Dot",
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	251 Tok.Assign:"Assign",
74 192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....) johnsen@johnsen-laptop parents: 72 diff changeset	252 Tok.Plus:"Plus",
126 c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	253 Tok.PlusAssign:"PlusAssign",
74 192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....) johnsen@johnsen-laptop parents: 72 diff changeset	254 Tok.Minus:"Minus",
126 c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	255 Tok.MinusAssign:"MinusAssign",
74 192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....) johnsen@johnsen-laptop parents: 72 diff changeset	256 Tok.Star:"Star",
126 c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	257 Tok.StarAssign:"StarAssign",
74 192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....) johnsen@johnsen-laptop parents: 72 diff changeset	258 Tok.Slash:"Slash",
126 c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	259 Tok.SlashAssign:"SlashAssign",
74 192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....) johnsen@johnsen-laptop parents: 72 diff changeset	260 Tok.Percent:"Percent",
126 c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	261 Tok.PercentAssign:"PercentAssign",
123 6a5f745d351c Parsing <<, >> and >>>. Anders Johnsen <skabet@gmail.com> parents: 104 diff changeset	262 Tok.LeftShift:"LeftShift",
6a5f745d351c Parsing <<, >> and >>>. Anders Johnsen <skabet@gmail.com> parents: 104 diff changeset	263 Tok.RightShift:"RightShift",
6a5f745d351c Parsing <<, >> and >>>. Anders Johnsen <skabet@gmail.com> parents: 104 diff changeset	264 Tok.UnsignedRightShift:"UnsignedRightShift",
5 2c5a8f4c254a Added very simple if support. Anders Halager <halager@gmail.com> parents: 1 diff changeset	265 Tok.Integer:"Integer",
2c5a8f4c254a Added very simple if support. Anders Halager <halager@gmail.com> parents: 1 diff changeset	266 Tok.If:"If",
11 642c6a998fd9 Support for while statements and fixed scope for if Anders Halager <halager@gmail.com> parents: 8 diff changeset	267 Tok.While:"While",
36 ce17bea8e9bd Switch statements support Anders Halager <halager@gmail.com> parents: 28 diff changeset	268 Tok.Switch:"Switch",
ce17bea8e9bd Switch statements support Anders Halager <halager@gmail.com> parents: 28 diff changeset	269 Tok.Case:"Case",
ce17bea8e9bd Switch statements support Anders Halager <halager@gmail.com> parents: 28 diff changeset	270 Tok.Default:"Default",
21 0fb2d13dce37 Now working with gdc also (gdc use reverse paremeter validating on function calls) johnsen@johnsen-laptop parents: 11 diff changeset	271 Tok.Comma:"Comma",
5 2c5a8f4c254a Added very simple if support. Anders Halager <halager@gmail.com> parents: 1 diff changeset	272 Tok.Return:"Return",
22 e331e4e816e4 now handling structs to some extend johnsen@johnsen-laptop parents: 21 diff changeset	273 Tok.Struct:"Struct",
36 ce17bea8e9bd Switch statements support Anders Halager <halager@gmail.com> parents: 28 diff changeset	274 Tok.Colon:"Colon",
68 381975d76baf A LOT of bug fixing - also implemented implicit casts. If you do a --ast-dump-code on a target with some algebra of differant types, you should now see the type casts being made. Also, Tests are again back with only switches failing... Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	275 Tok.Seperator:"Seperator",
94 48bb2287c035 Added Modules. Right now it's very simple - will grow with time and need. Anders Johnsen <skabet@gmail.com> parents: 88 diff changeset	276 Tok.Cast:"Cast",
48bb2287c035 Added Modules. Right now it's very simple - will grow with time and need. Anders Johnsen <skabet@gmail.com> parents: 88 diff changeset	277 Tok.Module:"Module",
104 7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST. Anders Johnsen <skabet@gmail.com> parents: 103 diff changeset	278 Tok.Import:"Import",
126 c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	279 Tok.String:"String",
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	280 Tok.Public:"Public",
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	281 Tok.Private:"Private",
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	282 Tok.Protected:"Protected",
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	283 Tok.Package:"Package",
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	284 Tok.Export:"Export",
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	285 Tok.Static:"Static",
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	286 Tok.Final:"Finale",
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	287 Tok.Public:"Public",
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	288 Tok.Const:"Const",
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	289 Tok.Abstract:"Abstract",
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	290 Tok.Override:"Override",
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	291 Tok.Depracted:"Depracted",
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet). Anders Johnsen <skabet@gmail.com> parents: 123 diff changeset	292 Tok.Auto:"Auto"
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	293 ];
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	294 }

Mercurial > projects > dang

annotate lexer/Token.d @ 126:c3b24e7e8cf8