Mercurial > projects > dang
annotate lexer/Token.d @ 42:4e879f82dd64 new_gen
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
| author | Anders Johnsen <skabet@gmail.com> |
|---|---|
| date | Tue, 22 Apr 2008 22:25:07 +0200 |
| parents | 858b9805843d |
| children | 495188f9078e |
| rev | line source |
|---|---|
| 1 | 1 module lexer.Token; |
| 2 | |
| 3 public | |
| 4 import misc.Location; | |
| 5 | |
| 6 import Integer = tango.text.convert.Integer; | |
| 7 | |
|
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
8 /** |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
9 The Token struct will be used through the Lexer, Parser and other |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
10 modules as a location into source. |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
11 |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
12 The Token should always be optimized for size to limit unnecessary |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
13 memory usage. |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
14 */ |
| 1 | 15 struct Token |
| 16 { | |
| 17 Tok type; | |
| 18 Location location; | |
| 19 uint length; | |
| 20 | |
|
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
21 /** |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
22 Create a new token with a Tok type, Location in source and a |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
23 length of how many chars the Token span in the source |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
24 */ |
| 1 | 25 static Token opCall (Tok type, Location location, uint length) |
| 26 { | |
| 27 Token t; | |
| 28 t.type = type; | |
| 29 t.location = location; | |
| 30 t.length = length; | |
| 31 return t; | |
| 32 } | |
| 33 | |
|
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
34 /** |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
35 Get the type of the Token as a string |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
36 */ |
| 1 | 37 char[] getType () |
| 38 { | |
| 39 return typeToString[this.type]; | |
| 40 } | |
| 41 | |
|
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
42 /** |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
43 A human readable dump of a Token |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
44 */ |
| 1 | 45 char[] toString () |
| 46 { | |
| 47 return this.getType()~": Len: "~Integer.toString(this.length) | |
| 48 ~", Loc: "~location.toString; | |
| 49 } | |
| 50 | |
|
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
51 /** |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
52 Get the string in the source that matches what this Token is |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
53 covering. |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
54 */ |
| 1 | 55 char[] get () |
| 56 { | |
| 57 return location.get(length); | |
| 58 } | |
| 59 } | |
| 60 | |
|
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
61 /** |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
62 Tok is short for TokenType. This enum list is to supply the Token |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
63 with a type. |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
64 |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
65 This enum is used to switch over "many" places. |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
66 */ |
| 1 | 67 enum Tok : ushort |
| 68 { | |
| 69 /* Non-code related tokens */ | |
| 70 EOF, | |
| 71 | |
| 72 /* Basic types */ | |
| 73 Identifier, | |
| 74 Integer, | |
| 75 | |
| 76 /* Basic operators */ | |
| 77 Assign, | |
| 78 Add, Sub, | |
| 79 Mul, Div, | |
| 80 Comma, | |
| 81 | |
| 82 /* Symbols */ | |
| 83 OpenParentheses, | |
| 84 CloseParentheses, | |
| 85 OpenBrace, | |
| 86 CloseBrace, | |
| 87 Seperator, | |
|
36
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
88 Colon, |
|
28
69464d465284
Now supporting structs - both read and write. Still a few errors though, so watch out.
Anders Johnsen <skabet@gmail.com>
parents:
22
diff
changeset
|
89 Dot, |
| 1 | 90 |
|
8
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
91 /* Comparator operators */ |
|
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
92 Eq, Ne, |
|
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
93 Lt, Gt, |
|
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
94 Le, Ge, |
|
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
95 |
|
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
96 Not, |
| 6 | 97 |
| 1 | 98 /* Keywords */ |
| 99 Byte, Ubyte, | |
| 100 Short, Ushort, | |
| 101 Int, Uint, | |
| 102 Long, Ulong, | |
| 103 | |
| 104 Float, Double, | |
| 105 | |
|
5
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
106 Bool, |
|
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
107 |
| 37 | 108 Void, |
| 109 | |
| 22 | 110 Struct, |
| 111 | |
|
11
642c6a998fd9
Support for while statements and fixed scope for if
Anders Halager <halager@gmail.com>
parents:
8
diff
changeset
|
112 If, Else, |
|
642c6a998fd9
Support for while statements and fixed scope for if
Anders Halager <halager@gmail.com>
parents:
8
diff
changeset
|
113 While, |
|
36
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
114 Switch, Case, Default, |
| 1 | 115 Return, |
| 116 | |
| 117 } | |
| 118 | |
|
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
119 /** |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
120 An associative array to supply a Tok to String function. |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
121 |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
122 Keep always this list updated when adding a new Tok. |
|
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
37
diff
changeset
|
123 */ |
| 1 | 124 public char[][Tok] typeToString; |
| 125 | |
| 126 static this() | |
| 127 { | |
| 128 typeToString = | |
| 129 [ | |
| 130 Tok.EOF:"EOF"[], | |
| 131 Tok.Identifier:"Identifier", | |
| 132 Tok.Byte:"Byte", | |
| 133 Tok.Short:"Short", | |
| 134 Tok.Int:"Int", | |
| 135 Tok.Long:"Long", | |
|
5
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
136 Tok.Bool:"Bool", |
| 37 | 137 Tok.Void:"Void", |
|
8
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
138 Tok.Eq:"Eq", |
|
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
139 Tok.Ne:"Ne", |
|
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
140 Tok.Lt:"Lt", |
|
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
141 Tok.Le:"Le", |
|
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
142 Tok.Gt:"Gt", |
|
2e1069ee21af
Added Ne, Lt, Le, Gt, Ge and Not in lexer
johnsen@johnsen-desktop
parents:
6
diff
changeset
|
143 Tok.Ge:"Ge", |
| 1 | 144 Tok.OpenParentheses:"OpenParentheses", |
| 145 Tok.CloseParentheses:"CloseParentheses", | |
| 146 Tok.OpenBrace:"OpenBrace", | |
| 147 Tok.CloseBrace:"CloseBrace", | |
|
28
69464d465284
Now supporting structs - both read and write. Still a few errors though, so watch out.
Anders Johnsen <skabet@gmail.com>
parents:
22
diff
changeset
|
148 Tok.Dot:"Dot", |
| 1 | 149 Tok.Assign:"Assign", |
| 150 Tok.Add:"Add", | |
| 151 Tok.Sub:"Sub", | |
| 152 Tok.Mul:"Mul", | |
| 153 Tok.Div:"Div", | |
|
5
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
154 Tok.Integer:"Integer", |
|
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
155 Tok.If:"If", |
|
11
642c6a998fd9
Support for while statements and fixed scope for if
Anders Halager <halager@gmail.com>
parents:
8
diff
changeset
|
156 Tok.While:"While", |
|
36
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
157 Tok.Switch:"Switch", |
|
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
158 Tok.Case:"Case", |
|
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
159 Tok.Default:"Default", |
|
21
0fb2d13dce37
Now working with gdc also (gdc use reverse paremeter validating on function calls)
johnsen@johnsen-laptop
parents:
11
diff
changeset
|
160 Tok.Comma:"Comma", |
|
5
2c5a8f4c254a
Added very simple if support.
Anders Halager <halager@gmail.com>
parents:
1
diff
changeset
|
161 Tok.Return:"Return", |
| 22 | 162 Tok.Struct:"Struct", |
|
36
ce17bea8e9bd
Switch statements support
Anders Halager <halager@gmail.com>
parents:
28
diff
changeset
|
163 Tok.Colon:"Colon", |
| 1 | 164 Tok.Seperator:"Seperator" |
| 165 ]; | |
| 166 } |