Mercurial > projects > dang
annotate lexer/Token.d @ 94:48bb2287c035 new_gen
Added Modules. Right now it's very simple - will grow with time and need.
author | Anders Johnsen <skabet@gmail.com> |
---|---|
date | Tue, 06 May 2008 16:24:14 +0200 |
parents | eb5b2c719a39 |
children | 09b4d74cb3f5 |
rev | line source |
---|---|
1 | 1 module lexer.Token; |
2 | |
3 public | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
4 import basic.SourceLocation; |
1 | 5 |
6 import Integer = tango.text.convert.Integer; | |
7 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
8 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
9 The Token struct will be used through the Lexer, Parser and other |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
10 modules as a location into source. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
11 |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
12 The Token should always be optimized for size to limit unnecessary |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
13 memory usage. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
14 */ |
1 | 15 struct Token |
16 { | |
17 Tok type; | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
18 SLoc location; |
1 | 19 uint length; |
20 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
21 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
22 Create a new token with a Tok type, Location in source and a |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
23 length of how many chars the Token span in the source |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
24 */ |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
25 static Token opCall (Tok type, SLoc location, uint length) |
1 | 26 { |
27 Token t; | |
28 t.type = type; | |
29 t.location = location; | |
30 t.length = length; | |
31 return t; | |
32 } | |
33 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
34 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
35 Get the type of the Token as a string |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
36 */ |
1 | 37 char[] getType () |
38 { | |
39 return typeToString[this.type]; | |
40 } | |
41 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
42 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
43 A human readable dump of a Token |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
44 */ |
1 | 45 char[] toString () |
46 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
47 return this.getType()~": Len: "~Integer.toString(this.length); |
1 | 48 } |
49 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
50 /// Get the range of this token |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
51 SourceRange asRange() { return SourceRange(location, location + length); } |
44
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
52 |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
53 /** |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
54 Returns true if the type of this token is a basic type (int, float, ...). |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
55 Void is included, although a void in it self is not really a type. |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
56 */ |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
57 bool isBasicType() |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
58 { |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
59 return type >= Tok.Byte && type <= Tok.Void; |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
60 } |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
61 |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
62 /** |
48
b6c1dc30ca4b
Only tests that dont pass now are structs and switches
Anders Halager <halager@gmail.com>
parents:
44
diff
changeset
|
63 Returns true for all the various assignments (=, +=, *= ...) |
b6c1dc30ca4b
Only tests that dont pass now are structs and switches
Anders Halager <halager@gmail.com>
parents:
44
diff
changeset
|
64 */ |
b6c1dc30ca4b
Only tests that dont pass now are structs and switches
Anders Halager <halager@gmail.com>
parents:
44
diff
changeset
|
65 bool isAssignment() |
b6c1dc30ca4b
Only tests that dont pass now are structs and switches
Anders Halager <halager@gmail.com>
parents:
44
diff
changeset
|
66 { |
b6c1dc30ca4b
Only tests that dont pass now are structs and switches
Anders Halager <halager@gmail.com>
parents:
44
diff
changeset
|
67 return type == Tok.Assign; |
b6c1dc30ca4b
Only tests that dont pass now are structs and switches
Anders Halager <halager@gmail.com>
parents:
44
diff
changeset
|
68 } |
b6c1dc30ca4b
Only tests that dont pass now are structs and switches
Anders Halager <halager@gmail.com>
parents:
44
diff
changeset
|
69 |
b6c1dc30ca4b
Only tests that dont pass now are structs and switches
Anders Halager <halager@gmail.com>
parents:
44
diff
changeset
|
70 /** |
44
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
71 Just a shortcut to avoid `token.type == Tok.Identifier`. |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
72 */ |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
73 bool isIdentifier() |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
74 { |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
75 return type == Tok.Identifier; |
495188f9078e
Big update - Moving towards a better, more seperated parser
Anders Halager <halager@gmail.com>
parents:
42
diff
changeset
|
76 } |
1 | 77 } |
78 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
79 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
80 Tok is short for TokenType. This enum list is to supply the Token |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
81 with a type. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
82 |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
83 This enum is used to switch over "many" places. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
84 */ |
1 | 85 enum Tok : ushort |
86 { | |
87 /* Non-code related tokens */ | |
88 EOF, | |
89 | |
90 /* Basic types */ | |
91 Identifier, | |
92 Integer, | |
93 | |
94 /* Basic operators */ | |
95 Assign, | |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
96 Plus, Minus, |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
97 Star, Slash, Percent, |
1 | 98 Comma, |
99 | |
100 /* Symbols */ | |
101 OpenParentheses, | |
102 CloseParentheses, | |
103 OpenBrace, | |
104 CloseBrace, | |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
68
diff
changeset
|
105 OpenBracket, |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
68
diff
changeset
|
106 CloseBracket, |
1 | 107 Seperator, |
36
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
108 Colon, |
28
69464d465284
Now supporting structs - both read and write. Still a few errors though, so watch out.
Anders Johnsen <skabet@gmail.com>
parents:
22
diff
changeset
|
109 Dot, |
1 | 110 |
8
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
111 /* Comparator operators */ |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
112 Eq, Ne, |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
113 Lt, Gt, |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
114 Le, Ge, |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
115 |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
116 Not, |
6 | 117 |
1 | 118 /* Keywords */ |
119 Byte, Ubyte, | |
120 Short, Ushort, | |
121 Int, Uint, | |
122 Long, Ulong, | |
123 | |
124 Float, Double, | |
125 | |
5
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
126 Bool, |
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
127 |
37 | 128 Void, |
129 | |
22 | 130 Struct, |
131 | |
11
642c6a998fd9
Support for while statements and fixed scope for if
Anders Halager <halager@gmail.com>
parents:
8
diff
changeset
|
132 If, Else, |
642c6a998fd9
Support for while statements and fixed scope for if
Anders Halager <halager@gmail.com>
parents:
8
diff
changeset
|
133 While, |
36
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
134 Switch, Case, Default, |
68
381975d76baf
A LOT of bug fixing - also implemented implicit casts. If you do a --ast-dump-code on a target with some algebra of differant types, you should now see the type casts being made. Also, Tests are again back with only switches failing...
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
135 Return, Cast, |
1 | 136 |
94
48bb2287c035
Added Modules. Right now it's very simple - will grow with time and need.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
137 Module, Import, |
48bb2287c035
Added Modules. Right now it's very simple - will grow with time and need.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
138 |
1 | 139 } |
140 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
141 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
142 An associative array to supply a Tok to String function. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
143 |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
144 Keep always this list updated when adding a new Tok. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
145 */ |
1 | 146 public char[][Tok] typeToString; |
147 | |
148 static this() | |
149 { | |
150 typeToString = | |
151 [ | |
152 Tok.EOF:"EOF"[], | |
153 Tok.Identifier:"Identifier", | |
154 Tok.Byte:"Byte", | |
155 Tok.Short:"Short", | |
156 Tok.Int:"Int", | |
157 Tok.Long:"Long", | |
5
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
158 Tok.Bool:"Bool", |
37 | 159 Tok.Void:"Void", |
8
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
160 Tok.Eq:"Eq", |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
161 Tok.Ne:"Ne", |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
162 Tok.Lt:"Lt", |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
163 Tok.Le:"Le", |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
164 Tok.Gt:"Gt", |
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
165 Tok.Ge:"Ge", |
1 | 166 Tok.OpenParentheses:"OpenParentheses", |
167 Tok.CloseParentheses:"CloseParentheses", | |
168 Tok.OpenBrace:"OpenBrace", | |
169 Tok.CloseBrace:"CloseBrace", | |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
68
diff
changeset
|
170 Tok.OpenBracket:"OpenBracket", |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
68
diff
changeset
|
171 Tok.CloseBracket:"CloseBracket", |
28
69464d465284
Now supporting structs - both read and write. Still a few errors though, so watch out.
Anders Johnsen <skabet@gmail.com>
parents:
22
diff
changeset
|
172 Tok.Dot:"Dot", |
1 | 173 Tok.Assign:"Assign", |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
174 Tok.Plus:"Plus", |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
175 Tok.Minus:"Minus", |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
176 Tok.Star:"Star", |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
177 Tok.Slash:"Slash", |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
178 Tok.Percent:"Percent", |
5
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
179 Tok.Integer:"Integer", |
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
180 Tok.If:"If", |
11
642c6a998fd9
Support for while statements and fixed scope for if
Anders Halager <halager@gmail.com>
parents:
8
diff
changeset
|
181 Tok.While:"While", |
36
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
182 Tok.Switch:"Switch", |
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
183 Tok.Case:"Case", |
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
184 Tok.Default:"Default", |
21
0fb2d13dce37
Now working with gdc also (gdc use reverse paremeter validating on function calls)
johnsen@johnsen-laptop
parents:
11
diff
changeset
|
185 Tok.Comma:"Comma", |
5
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
186 Tok.Return:"Return", |
22 | 187 Tok.Struct:"Struct", |
36
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
188 Tok.Colon:"Colon", |
68
381975d76baf
A LOT of bug fixing - also implemented implicit casts. If you do a --ast-dump-code on a target with some algebra of differant types, you should now see the type casts being made. Also, Tests are again back with only switches failing...
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
189 Tok.Seperator:"Seperator", |
94
48bb2287c035
Added Modules. Right now it's very simple - will grow with time and need.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
190 Tok.Cast:"Cast", |
48bb2287c035
Added Modules. Right now it's very simple - will grow with time and need.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
191 Tok.Module:"Module", |
48bb2287c035
Added Modules. Right now it's very simple - will grow with time and need.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
192 Tok.Import:"Import" |
1 | 193 ]; |
194 } |