Mercurial > projects > dang
annotate lexer/Lexer.d @ 126:c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
author | Anders Johnsen <skabet@gmail.com> |
---|---|
date | Tue, 27 May 2008 10:32:31 +0200 |
parents | 6a5f745d351c |
children | 2be29b296081 |
rev | line source |
---|---|
1 | 1 module lexer.Lexer; |
2 | |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
3 import basic.Message, |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
4 basic.SourceManager; |
1 | 5 |
6 import lexer.Token, | |
7 lexer.Keyword; | |
8 | |
9 import tango.io.Stdout; | |
10 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
11 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
12 The Lexer class will supply you with methods to tokenize a D file. Supply the |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
13 Lexer with a DataSource and you can 'peek' and 'next' Tokens from the file. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
14 |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
15 For more info about Tokens, look up the lexer.Token module. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
16 */ |
1 | 17 class Lexer |
18 { | |
19 public: | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
20 |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
21 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
22 Create a new Lexer. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
23 */ |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
24 this(SourceLocation start, SourceManager src_mgr, MessageHandler messages) |
1 | 25 { |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
26 this.messages = messages; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
27 sm = src_mgr; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
28 start_loc = start; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
29 position = 0; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
30 source = sm.getRawData(start_loc); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
31 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
32 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
33 charTable.length = 256; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
34 foreach (c; "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
35 charTable[c] = CharType.Letter; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
36 |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
37 foreach (c; "0123456789") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
38 charTable[c] = CharType.Number; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
39 |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
40 foreach (c; "(){}[];:.,=!<>+-*/%\"`") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
41 charTable[c] = CharType.Symbol; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
42 |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
43 foreach (c; " \n") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
44 charTable[c] = CharType.Whitespace; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
45 |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
46 foreach (c; "'\\") |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
47 charTable[c] = CharType.Other; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
48 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
49 symbolFunctions.length = 256; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
50 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
51 symbolFunctions['('] = &openParentheses; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
52 symbolFunctions[')'] = &closeParentheses; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
53 symbolFunctions['{'] = &openBrace; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
54 symbolFunctions['}'] = &closeBrace; |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
55 symbolFunctions['['] = &openBracket; |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
56 symbolFunctions[']'] = &closeBracket; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
57 symbolFunctions[';'] = &seperator; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
58 symbolFunctions[':'] = : |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
59 symbolFunctions['.'] = ˙ |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
60 symbolFunctions[','] = , |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
61 symbolFunctions['='] = &eq; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
62 symbolFunctions['!'] = ≠ |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
63 symbolFunctions['<'] = ≤ |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
64 symbolFunctions['>'] = ≥ |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
65 symbolFunctions['+'] = + |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
66 symbolFunctions['-'] = − |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
67 symbolFunctions['*'] = ☆ |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
68 symbolFunctions['/'] = &slash; |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
69 symbolFunctions['%'] = &percent; |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
70 symbolFunctions['"'] = &string; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
71 symbolFunctions['`'] = &string; |
1 | 72 } |
73 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
74 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
75 Get the next token from the source. This method will move the |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
76 internal position forward to the next Token. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
77 |
43
a712c530b7cc
A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents:
42
diff
changeset
|
78 return: A Token - Token.type is TokType.EOF if there is |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
79 no more tokens in the file. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
80 */ |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
81 Token next() |
1 | 82 { |
83 switch (getNextChar) | |
84 { | |
85 case CharType.EOF: | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
86 SLoc loc; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
87 return Token(Tok.EOF, loc, 0); |
1 | 88 |
89 case CharType.Whitespace: | |
90 position += 1; | |
91 return this.next; | |
92 | |
93 case CharType.Symbol: | |
94 return lexSymbol; | |
95 | |
96 case CharType.Letter: | |
97 return lexLetter; | |
98 | |
99 case CharType.Number: | |
100 return lexNumber; | |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
101 case CharType.Other: |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
102 messages.report(UnexpectedTok, Loc(position)).fatal(ExitLevel.Lexer); |
1 | 103 } |
104 } | |
105 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
106 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
107 Get the next token from the source. This method will NOT move the |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
108 internal position forward, and thereby having no side-effects. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
109 |
43
a712c530b7cc
A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents:
42
diff
changeset
|
110 return: A Token - Token.type is TokType.EOF if there is |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
111 no more tokens in the file. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
112 */ |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
113 Token peek(int skip = 0) |
1 | 114 { |
115 int oldPosition = this.position; | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
116 while (skip-- > 0) |
1 | 117 this.next; |
118 Token t = this.next; | |
119 this.position = oldPosition; | |
120 return t; | |
121 } | |
122 | |
123 private: | |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
124 Token eq() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
125 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
126 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
127 return Token(Tok.Eq, Loc(position++ - 1), 2); |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
128 return Token(Tok.Assign, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
129 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
130 Token openBrace() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
131 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
132 return Token(Tok.OpenBrace, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
133 } |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
134 Token closeBrace() |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
135 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
136 return Token(Tok.CloseBrace, Loc(position - 1), 1); |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
137 } |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
138 Token openParentheses() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
139 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
140 return Token(Tok.OpenParentheses, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
141 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
142 Token closeParentheses() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
143 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
144 return Token(Tok.CloseParentheses, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
145 } |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
146 Token openBracket() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
147 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
148 return Token(Tok.OpenBracket, Loc(position - 1), 1); |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
149 } |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
150 Token closeBracket() |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
151 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
152 return Token(Tok.CloseBracket, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
153 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
154 Token seperator() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
155 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
156 return Token(Tok.Seperator, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
157 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
158 Token colon() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
159 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
160 return Token(Tok.Colon, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
161 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
162 Token dot() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
163 { |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
164 int pos = 0; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
165 while(getNextChar(0) == CharType.Number || |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
166 this.source[position + pos + 1] == '_') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
167 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
168 if(getNextChar(0) == CharType.Number) |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
169 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
170 position--; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
171 return lexNumber(); |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
172 } |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
173 pos++; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
174 } |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
175 return Token(Tok.Dot, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
176 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
177 Token comma() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
178 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
179 return Token(Tok.Comma, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
180 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
181 Token ne() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
182 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
183 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
184 return Token(Tok.Ne, Loc(position++ - 1), 2); |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
185 return Token(Tok.Not, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
186 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
187 Token le() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
188 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
189 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
190 return Token(Tok.Le, Loc(position++ - 1), 2); |
123 | 191 if(source[position] == '<') |
192 return Token(Tok.LeftShift, Loc(position++ - 1), 2); | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
193 return Token(Tok.Lt, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
194 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
195 Token ge() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
196 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
197 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
198 return Token(Tok.Ge, Loc(position++ - 1), 2); |
123 | 199 if(source[position] == '>') |
200 if(source[position+1] == '>') | |
201 { | |
202 position += 2; | |
203 return Token(Tok.UnsignedRightShift, Loc(position - 1), 3); | |
204 } | |
205 else | |
206 return Token(Tok.RightShift, Loc(position++ - 1), 2); | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
207 return Token(Tok.Gt, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
208 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
209 Token plus() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
210 { |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
211 if(source[position] == '=') |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
212 return Token(Tok.PlusAssign, Loc(position++ - 1), 2); |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
213 return Token(Tok.Plus, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
214 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
215 Token minus() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
216 { |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
217 if(source[position] == '=') |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
218 return Token(Tok.MinusAssign, Loc(position++ - 1), 2); |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
219 return Token(Tok.Minus, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
220 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
221 Token star() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
222 { |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
223 if(source[position] == '=') |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
224 return Token(Tok.StarAssign, Loc(position++ - 1), 2); |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
225 return Token(Tok.Star, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
226 } |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
227 Token slash() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
228 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
229 switch(source[position]) |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
230 { |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
231 case '=': |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
232 return Token(Tok.SlashAssign, Loc(position++ - 1), 2); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
233 case '/': |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
234 while(getNextChar != CharType.EOF) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
235 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
236 if(source[position++] == '\n') |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
237 return this.next; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
238 } |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
239 return Token(Tok.EOF, Loc(position), 0); |
1 | 240 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
241 case '*': |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
242 position += 2; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
243 while(getNextChar != CharType.EOF) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
244 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
245 ++position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
246 if(source[position-2] == '*') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
247 if(source[position-1] == '/') |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
248 { |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
249 return this.next; |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
250 } |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
251 } |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
252 messages.report(UnexpectedEOFBlock,Loc(position)); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
253 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
254 case '+': |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
255 position += 2; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
256 int nesting = 1; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
257 while(getNextChar != CharType.EOF) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
258 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
259 ++position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
260 if(source[position-2] == '+') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
261 if(source[position-1] == '/') |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
262 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
263 position++; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
264 nesting--; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
265 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
266 |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
267 if(source[position-2] == '/') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
268 if(source[position-1] == '+') |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
269 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
270 nesting++; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
271 position++; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
272 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
273 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
274 if(nesting == 0) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
275 return this.next; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
276 } |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
277 messages.report(UnexpectedEOFBlock,Loc(position)); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
278 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
279 default: |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
280 return Token(Tok.Slash, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
281 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
282 } |
62
78a6808b2e0f
Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents:
43
diff
changeset
|
283 |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
284 Token percent() |
62
78a6808b2e0f
Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents:
43
diff
changeset
|
285 { |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
286 if(source[position] == '=') |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
287 return Token(Tok.PercentAssign, Loc(position++ - 1), 2); |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
288 return Token(Tok.Percent, Loc(position - 1), 1); |
62
78a6808b2e0f
Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents:
43
diff
changeset
|
289 } |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
290 |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
291 Token string() |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
292 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
293 --position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
294 int start = position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
295 if(getNextChar() == CharType.Letter) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
296 position++; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
297 char end = '`'; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
298 switch(source[position]) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
299 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
300 case '"': |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
301 if(position > 0) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
302 if(source[position-1] == 'r') |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
303 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
304 end = '"'; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
305 goto string_wys; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
306 } |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
307 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
308 while(getNextChar != CharType.EOF) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
309 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
310 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
311 if (source[position-1] == '"' ) |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
312 { |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
313 if(getNextChar != CharType.EOF) |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
314 if (source[position] == 'c' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
315 source[position] == 'w' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
316 source[position] == 'd') |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
317 position++; |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
318 |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
319 return Token(Tok.String, Loc(start), position - start); |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
320 } |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
321 else if (source[position-1] == '\\') |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
322 position++; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
323 } |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
324 break; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
325 case '`': |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
326 string_wys: |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
327 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
328 while(getNextChar != CharType.EOF) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
329 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
330 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
331 if (source[position-1] == end ) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
332 return Token(Tok.String, Loc(start), position - start); |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
333 } |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
334 break; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
335 } |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
336 messages.report(UnexpectedEOFBlock, Loc(position)).fatal(ExitLevel.Lexer); |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
337 } |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
338 |
1 | 339 Token lexNumber () |
340 { | |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
341 bool sign; |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
342 |
1 | 343 int i = 0; |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
344 |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
345 |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
346 bool end = false; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
347 while(!end) |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
348 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
349 switch(getNextChar(i)) |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
350 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
351 case CharType.Number: |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
352 break; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
353 case CharType.Symbol: |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
354 if(this.source[position+i] == '.') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
355 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
356 break; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
357 } |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
358 if (this.source[position+i] == '+' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
359 this.source[position+i] == '-') |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
360 { |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
361 if (source[position+i-1] == 'e' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
362 source[position+i-1] == 'E') |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
363 break; |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
364 } |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
365 end = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
366 continue; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
367 case CharType.Letter: |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
368 if(this.source[position+i] == '_') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
369 break; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
370 if (this.source[position+i] == 'e' || |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
371 this.source[position+i] == 'E') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
372 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
373 break; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
374 } |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
375 end = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
376 continue; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
377 |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
378 default: |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
379 end = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
380 continue; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
381 } |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
382 i++; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
383 } |
1 | 384 |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
385 while(source[position+i] == 'u' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
386 source[position+i] == 'U' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
387 source[position+i] == 'L') |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
388 i += 1; |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
389 |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
390 |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
391 |
1 | 392 position += i; |
393 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
394 return Token(Tok.Integer, Loc(position - i), i); |
1 | 395 } |
396 | |
397 Token lexSymbol () | |
398 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
399 Token t = symbolFunctions[source[position++]](); |
1 | 400 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
401 return t; |
1 | 402 } |
403 | |
404 Token lexLetter () | |
405 { | |
406 int i = 0; | |
407 bool hasNumber = false; | |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
408 if (source[position+1] == '"' || |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
409 source[position+1] == '`') |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
410 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
411 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
412 return string; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
413 } |
1 | 414 while (getNextChar(++i) == CharType.Letter || |
415 getNextChar(i) == CharType.Number) | |
416 { | |
417 if (getNextChar(i) == CharType.Number) | |
418 { | |
419 hasNumber = true; | |
420 } | |
421 } | |
422 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
423 Token t = Token(Tok.Identifier, Loc(), i); |
1 | 424 |
425 if (!hasNumber) | |
426 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
427 char[] str = source[position .. position + i]; |
1 | 428 if(str in keywords) |
429 t.type = keywords[str]; | |
430 } | |
431 | |
432 position += i; | |
433 | |
434 return t; | |
435 } | |
436 | |
437 CharType getNextChar(int offset = 0) | |
438 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
439 if (position + offset >= this.source.length) |
1 | 440 return CharType.EOF; |
441 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
442 char current = source[position + offset]; |
1 | 443 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
444 CharType c = charTable[current]; |
1 | 445 |
43
a712c530b7cc
A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents:
42
diff
changeset
|
446 if(c == CharType.INVALID) |
92
771ac63898e2
A few better parser errors plus renaming most of the sema classes to match that they do now. Some have changes a lot.
Anders Johnsen <skabet@gmail.com>
parents:
91
diff
changeset
|
447 messages.report(InvalidSymbol, Loc()) |
771ac63898e2
A few better parser errors plus renaming most of the sema classes to match that they do now. Some have changes a lot.
Anders Johnsen <skabet@gmail.com>
parents:
91
diff
changeset
|
448 .arg(Integer.toString(cast(int)current)) |
771ac63898e2
A few better parser errors plus renaming most of the sema classes to match that they do now. Some have changes a lot.
Anders Johnsen <skabet@gmail.com>
parents:
91
diff
changeset
|
449 .fatal(ExitLevel.Lexer); |
1 | 450 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
451 return c; |
1 | 452 |
453 } | |
454 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
455 private final SourceLocation Loc(int pos = -1) |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
456 { |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
457 if (pos < 0) |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
458 return start_loc + position; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
459 return start_loc + pos; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
460 } |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
461 |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
462 SourceManager sm; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
463 SourceLocation start_loc; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
464 int position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
465 char[] source; |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
466 MessageHandler messages; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
467 CharType[] charTable; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
468 Token delegate()[] symbolFunctions; |
1 | 469 } |
470 | |
471 enum CharType : ubyte | |
472 { | |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
473 INVALID, |
1 | 474 Letter, |
475 Number, | |
476 Symbol, | |
477 Whitespace, | |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
478 Other, |
1 | 479 |
480 EOF | |
481 } | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
482 |