Mercurial > projects > dang
annotate lexer/Token.d @ 172:01c2c49775ef
- Changed Parser to be more clean on type parsing.
- Parsing int function(int), and the like, types.(Function pointers)
- Fixed a design fault that made Symbol be wrong. Now Symbols are created with a factory with the help of types.
author | Anders Johnsen <skabet@gmail.com> |
---|---|
date | Thu, 24 Jul 2008 20:31:24 +0200 |
parents | 57b0b4464a0b |
children | dc9bf56b7ace |
rev | line source |
---|---|
1 | 1 module lexer.Token; |
2 | |
3 public | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
4 import basic.SourceLocation; |
1 | 5 |
6 import Integer = tango.text.convert.Integer; | |
7 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
8 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
9 The Token struct will be used through the Lexer, Parser and other |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
10 modules as a location into source. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
11 |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
12 The Token should always be optimized for size to limit unnecessary |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
13 memory usage. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
14 */ |
1 | 15 struct Token |
16 { | |
17 Tok type; | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
18 SLoc location; |
1 | 19 uint length; |
20 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
21 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
22 Create a new token with a Tok type, Location in source and a |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
23 length of how many chars the Token span in the source |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
24 */ |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
25 static Token opCall (Tok type, SLoc location, uint length) |
1 | 26 { |
27 Token t; | |
28 t.type = type; | |
29 t.location = location; | |
30 t.length = length; | |
31 return t; | |
32 } | |
33 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
34 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
35 Get the type of the Token as a string |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
36 */ |
1 | 37 char[] getType () |
38 { | |
39 return typeToString[this.type]; | |
40 } | |
41 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
42 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
43 A human readable dump of a Token |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
44 */ |
1 | 45 char[] toString () |
46 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
47 return this.getType()~": Len: "~Integer.toString(this.length); |
1 | 48 } |
49 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
50 /// Get the range of this token |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
51 SourceRange asRange() { return SourceRange(location, location + length); } |
44
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
52 |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
53 /** |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
54 Returns true if the type of this token is a basic type (int, float, ...). |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
55 Void is included, although a void in it self is not really a type. |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
56 */ |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
57 bool isBasicType() |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
58 { |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
59 return type >= Tok.Byte && type <= Tok.Void; |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
60 } |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
61 |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
62 /** |
48
b6c1dc30ca4b
Only tests that dont pass now are structs and switches
Anders Halager <halager@gmail.com>
parents:
44
diff
changeset
|
63 Returns true for all the various assignments (=, +=, *= ...) |
b6c1dc30ca4b
Only tests that dont pass now are structs and switches
Anders Halager <halager@gmail.com>
parents:
44
diff
changeset
|
64 */ |
b6c1dc30ca4b
Only tests that dont pass now are structs and switches
Anders Halager <halager@gmail.com>
parents:
44
diff
changeset
|
65 bool isAssignment() |
b6c1dc30ca4b
Only tests that dont pass now are structs and switches
Anders Halager <halager@gmail.com>
parents:
44
diff
changeset
|
66 { |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
67 return type >= Tok.Assign && type <= Tok.PercentAssign; |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
68 } |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
69 |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
70 /** |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
71 Returns true for all attributes( public, static, private...) |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
72 */ |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
73 bool isAttribute() |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
74 { |
140
927ae00bd9d2
Added support for extern keyword. Being ignored atm though. Also changed ast/Module, so that you can get a list of only vars, functions or structs.
Anders Johnsen <skabet@gmail.com>
parents:
136
diff
changeset
|
75 return type >= Tok.Public && type <= Tok.Extern; |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
76 } |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
77 |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
78 /** |
136 | 79 Returns true for all attributes( public, static, private...) |
80 */ | |
81 bool isBaseClassProtection() | |
82 { | |
83 return type >= Tok.Public && type <= Tok.Export; | |
84 } | |
85 | |
86 /** | |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
87 just a shortcut to avoid `token.type == tok.Switch`. |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
88 */ |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
89 bool isSwitch() |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
90 { |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
91 return type == Tok.Switch; |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
92 } |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
93 |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
94 /** |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
95 just a shortcut to avoid `token.type == tok.While`. |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
96 */ |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
97 bool isWhile() |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
98 { |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
99 return type == Tok.While; |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
100 } |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
101 |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
102 /** |
146 | 103 just a shortcut to avoid `token.type == tok.For`. |
104 */ | |
105 bool isFor() | |
106 { | |
107 return type == Tok.For; | |
108 } | |
109 | |
110 /** | |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
111 just a shortcut to avoid `token.type == tok.If`. |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
112 */ |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
113 bool isIf() |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
114 { |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
115 return type == Tok.If; |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
116 } |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
117 |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
118 /** |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
119 just a shortcut to avoid `token.type == tok.Return`. |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
120 */ |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
121 bool isReturn() |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
122 { |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
123 return type == Tok.Return; |
48
b6c1dc30ca4b
Only tests that dont pass now are structs and switches
Anders Halager <halager@gmail.com>
parents:
44
diff
changeset
|
124 } |
b6c1dc30ca4b
Only tests that dont pass now are structs and switches
Anders Halager <halager@gmail.com>
parents:
44
diff
changeset
|
125 |
b6c1dc30ca4b
Only tests that dont pass now are structs and switches
Anders Halager <halager@gmail.com>
parents:
44
diff
changeset
|
126 /** |
44
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
127 Just a shortcut to avoid `token.type == Tok.Identifier`. |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
128 */ |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
129 bool isIdentifier() |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
130 { |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
131 return type == Tok.Identifier; |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
132 } |
1 | 133 } |
134 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
135 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
136 Tok is short for TokenType. This enum list is to supply the Token |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
137 with a type. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
138 |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
139 This enum is used to switch over "many" places. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
140 */ |
1 | 141 enum Tok : ushort |
142 { | |
143 /* Non-code related tokens */ | |
144 EOF, | |
145 | |
146 /* Basic types */ | |
147 Identifier, | |
148 Integer, | |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
149 String, |
1 | 150 |
151 /* Basic operators */ | |
152 Assign, | |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
153 PlusAssign, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
154 MinusAssign, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
155 StarAssign, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
156 SlashAssign, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
157 PercentAssign, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
158 Plus, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
159 Minus, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
160 Star, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
161 Slash, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
162 Percent, |
123 | 163 LeftShift, RightShift, UnsignedRightShift, |
1 | 164 Comma, |
165 | |
166 /* Symbols */ | |
167 OpenParentheses, | |
168 CloseParentheses, | |
169 OpenBrace, | |
170 CloseBrace, | |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
68
diff
changeset
|
171 OpenBracket, |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
68
diff
changeset
|
172 CloseBracket, |
1 | 173 Seperator, |
36
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
174 Colon, |
28
69464d465284
Now supporting structs - both read and write. Still a few errors though, so watch out.
Anders Johnsen <skabet@gmail.com>
parents:
22
diff
changeset
|
175 Dot, |
1 | 176 |
8
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
177 /* Comparator operators */ |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
178 Eq, Ne, |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
179 Lt, Gt, |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
180 Le, Ge, |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
181 |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
182 Not, |
6 | 183 |
1 | 184 /* Keywords */ |
185 Byte, Ubyte, | |
186 Short, Ushort, | |
187 Int, Uint, | |
188 Long, Ulong, | |
189 | |
103
09b4d74cb3f5
Parsing char, wchar and dchar as basic types.
Anders Johnsen <skabet@gmail.com>
parents:
94
diff
changeset
|
190 Char, Wchar, Dchar, |
09b4d74cb3f5
Parsing char, wchar and dchar as basic types.
Anders Johnsen <skabet@gmail.com>
parents:
94
diff
changeset
|
191 |
1 | 192 Float, Double, |
193 | |
5
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
194 Bool, |
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
195 |
37 | 196 Void, |
197 | |
154
0ea5d2f3e96b
Parsing "this" as constructor. Also removed regex from the test run program(seg fault - dmd???)
Anders Johnsen <skabet@gmail.com>
parents:
148
diff
changeset
|
198 Struct, Function, Delegate, Class, This, |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
199 Interface, Union, Typedef, Typeid, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
200 Typeof, Sizeof, Alias, |
22 | 201 |
11
642c6a998fd9
Support for while statements and fixed scope for if
Anders Halager <halager@gmail.com>
parents:
8
diff
changeset
|
202 If, Else, |
642c6a998fd9
Support for while statements and fixed scope for if
Anders Halager <halager@gmail.com>
parents:
8
diff
changeset
|
203 While, |
146 | 204 For, |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
205 Switch, Case, Default, Break, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
206 Return, Cast, |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
103
diff
changeset
|
207 |
94
48bb2287c035
Added Modules. Right now it's very simple - will grow with time and need.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
208 Module, Import, |
48bb2287c035
Added Modules. Right now it's very simple - will grow with time and need.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
209 |
158
57b0b4464a0b
Parsing "new", putting it in AST and performs some tests on it. Eg. if the contructor exists and the params matches.
Anders Johnsen <skabet@gmail.com>
parents:
154
diff
changeset
|
210 New, |
57b0b4464a0b
Parsing "new", putting it in AST and performs some tests on it. Eg. if the contructor exists and the params matches.
Anders Johnsen <skabet@gmail.com>
parents:
154
diff
changeset
|
211 |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
212 /* Attributes */ |
136 | 213 Public, Private, Package, Export, Protected, |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
214 Static, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
215 Final, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
216 Const, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
217 Abstract, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
218 Override, |
148
6ec686d9c87d
Fixed some for parsing, and removed a little ugly bug.
Anders Johnsen <skabet@gmail.com>
parents:
146
diff
changeset
|
219 Deprecated, |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
220 Auto, |
140
927ae00bd9d2
Added support for extern keyword. Being ignored atm though. Also changed ast/Module, so that you can get a list of only vars, functions or structs.
Anders Johnsen <skabet@gmail.com>
parents:
136
diff
changeset
|
221 Extern, |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
222 |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
223 Align, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
224 |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
225 Asm, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
226 |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
227 In, Out, Body, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
228 |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
229 Assert, Throw, Try, Catch, Finally, |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
230 |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
231 |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
232 |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
233 |
1 | 234 } |
235 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
236 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
237 An associative array to supply a Tok to String function. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
238 |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
239 Keep always this list updated when adding a new Tok. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
240 */ |
1 | 241 public char[][Tok] typeToString; |
242 | |
243 static this() | |
244 { | |
245 typeToString = | |
246 [ | |
247 Tok.EOF:"EOF"[], | |
248 Tok.Identifier:"Identifier", | |
249 Tok.Byte:"Byte", | |
250 Tok.Short:"Short", | |
251 Tok.Int:"Int", | |
252 Tok.Long:"Long", | |
103
09b4d74cb3f5
Parsing char, wchar and dchar as basic types.
Anders Johnsen <skabet@gmail.com>
parents:
94
diff
changeset
|
253 Tok.Char:"Char", |
09b4d74cb3f5
Parsing char, wchar and dchar as basic types.
Anders Johnsen <skabet@gmail.com>
parents:
94
diff
changeset
|
254 Tok.Wchar:"Wchar", |
09b4d74cb3f5
Parsing char, wchar and dchar as basic types.
Anders Johnsen <skabet@gmail.com>
parents:
94
diff
changeset
|
255 Tok.Dchar:"Dchar", |
5
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
256 Tok.Bool:"Bool", |
37 | 257 Tok.Void:"Void", |
172
01c2c49775ef
- Changed Parser to be more clean on type parsing.
Anders Johnsen <skabet@gmail.com>
parents:
158
diff
changeset
|
258 Tok.Function:"Function", |
8
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
259 Tok.Eq:"Eq", |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
260 Tok.Ne:"Ne", |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
261 Tok.Lt:"Lt", |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
262 Tok.Le:"Le", |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
263 Tok.Gt:"Gt", |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
264 Tok.Ge:"Ge", |
1 | 265 Tok.OpenParentheses:"OpenParentheses", |
266 Tok.CloseParentheses:"CloseParentheses", | |
267 Tok.OpenBrace:"OpenBrace", | |
268 Tok.CloseBrace:"CloseBrace", | |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
68
diff
changeset
|
269 Tok.OpenBracket:"OpenBracket", |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
68
diff
changeset
|
270 Tok.CloseBracket:"CloseBracket", |
28
69464d465284
Now supporting structs - both read and write. Still a few errors though, so watch out.
Anders Johnsen <skabet@gmail.com>
parents:
22
diff
changeset
|
271 Tok.Dot:"Dot", |
1 | 272 Tok.Assign:"Assign", |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
273 Tok.Plus:"Plus", |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
274 Tok.PlusAssign:"PlusAssign", |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
275 Tok.Minus:"Minus", |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
276 Tok.MinusAssign:"MinusAssign", |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
277 Tok.Star:"Star", |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
278 Tok.StarAssign:"StarAssign", |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
279 Tok.Slash:"Slash", |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
280 Tok.SlashAssign:"SlashAssign", |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
281 Tok.Percent:"Percent", |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
282 Tok.PercentAssign:"PercentAssign", |
123 | 283 Tok.LeftShift:"LeftShift", |
284 Tok.RightShift:"RightShift", | |
285 Tok.UnsignedRightShift:"UnsignedRightShift", | |
5
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
286 Tok.Integer:"Integer", |
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
287 Tok.If:"If", |
11
642c6a998fd9
Support for while statements and fixed scope for if
Anders Halager <halager@gmail.com>
parents:
8
diff
changeset
|
288 Tok.While:"While", |
146 | 289 Tok.For:"For", |
36
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
290 Tok.Switch:"Switch", |
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
291 Tok.Case:"Case", |
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
292 Tok.Default:"Default", |
21
0fb2d13dce37
Now working with gdc also (gdc use reverse paremeter validating on function calls)
johnsen@johnsen-laptop
parents:
11
diff
changeset
|
293 Tok.Comma:"Comma", |
5
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
294 Tok.Return:"Return", |
22 | 295 Tok.Struct:"Struct", |
154
0ea5d2f3e96b
Parsing "this" as constructor. Also removed regex from the test run program(seg fault - dmd???)
Anders Johnsen <skabet@gmail.com>
parents:
148
diff
changeset
|
296 Tok.Class:"Class", |
0ea5d2f3e96b
Parsing "this" as constructor. Also removed regex from the test run program(seg fault - dmd???)
Anders Johnsen <skabet@gmail.com>
parents:
148
diff
changeset
|
297 Tok.This:"This", |
36
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
298 Tok.Colon:"Colon", |
68
381975d76baf
A LOT of bug fixing - also implemented implicit casts. If you do a --ast-dump-code on a target with some algebra of differant types, you should now see the type casts being made. Also, Tests are again back with only switches failing...
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
299 Tok.Seperator:"Seperator", |
94
48bb2287c035
Added Modules. Right now it's very simple - will grow with time and need.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
300 Tok.Cast:"Cast", |
48bb2287c035
Added Modules. Right now it's very simple - will grow with time and need.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
301 Tok.Module:"Module", |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
103
diff
changeset
|
302 Tok.Import:"Import", |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
303 Tok.String:"String", |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
304 Tok.Public:"Public", |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
305 Tok.Private:"Private", |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
306 Tok.Protected:"Protected", |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
307 Tok.Package:"Package", |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
308 Tok.Export:"Export", |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
309 Tok.Static:"Static", |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
310 Tok.Final:"Finale", |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
311 Tok.Public:"Public", |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
312 Tok.Const:"Const", |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
313 Tok.Abstract:"Abstract", |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
314 Tok.Override:"Override", |
148
6ec686d9c87d
Fixed some for parsing, and removed a little ugly bug.
Anders Johnsen <skabet@gmail.com>
parents:
146
diff
changeset
|
315 Tok.Deprecated:"Deprecated", |
140
927ae00bd9d2
Added support for extern keyword. Being ignored atm though. Also changed ast/Module, so that you can get a list of only vars, functions or structs.
Anders Johnsen <skabet@gmail.com>
parents:
136
diff
changeset
|
316 Tok.Auto:"Auto", |
158
57b0b4464a0b
Parsing "new", putting it in AST and performs some tests on it. Eg. if the contructor exists and the params matches.
Anders Johnsen <skabet@gmail.com>
parents:
154
diff
changeset
|
317 Tok.Extern:"Extern", |
57b0b4464a0b
Parsing "new", putting it in AST and performs some tests on it. Eg. if the contructor exists and the params matches.
Anders Johnsen <skabet@gmail.com>
parents:
154
diff
changeset
|
318 Tok.New:"New" |
1 | 319 ]; |
320 } |