Mercurial > projects > dang
annotate lexer/Lexer.d @ 189:75d0544ddc45
Better error handling on unexpected EOF.
author | Anders Johnsen <skabet@gmail.com> |
---|---|
date | Fri, 25 Jul 2008 13:50:01 +0200 |
parents | e1e170c2cd44 |
children | 85e492318bb6 |
rev | line source |
---|---|
1 | 1 module lexer.Lexer; |
2 | |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
3 import basic.Message, |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
4 basic.SourceManager; |
1 | 5 |
6 import lexer.Token, | |
7 lexer.Keyword; | |
8 | |
9 import tango.io.Stdout; | |
10 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
11 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
12 The Lexer class will supply you with methods to tokenize a D file. Supply the |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
13 Lexer with a DataSource and you can 'peek' and 'next' Tokens from the file. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
14 |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
15 For more info about Tokens, look up the lexer.Token module. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
16 */ |
1 | 17 class Lexer |
18 { | |
19 public: | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
20 |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
21 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
22 Create a new Lexer. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
23 */ |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
24 this(SourceLocation start, SourceManager src_mgr, MessageHandler messages) |
1 | 25 { |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
26 this.messages = messages; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
27 sm = src_mgr; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
28 start_loc = start; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
29 position = 0; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
30 source = sm.getRawData(start_loc); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
31 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
32 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
33 charTable.length = 256; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
34 foreach (c; "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
35 charTable[c] = CharType.Letter; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
36 |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
37 foreach (c; "0123456789") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
38 charTable[c] = CharType.Number; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
39 |
176
dc9bf56b7ace
Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents:
154
diff
changeset
|
40 foreach (c; "(){}[];:.,=!<>+-*/%&\"`") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
41 charTable[c] = CharType.Symbol; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
42 |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
43 foreach (c; " \n") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
44 charTable[c] = CharType.Whitespace; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
45 |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
46 foreach (c; "'\\") |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
47 charTable[c] = CharType.Other; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
48 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
49 symbolFunctions.length = 256; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
50 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
51 symbolFunctions['('] = &openParentheses; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
52 symbolFunctions[')'] = &closeParentheses; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
53 symbolFunctions['{'] = &openBrace; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
54 symbolFunctions['}'] = &closeBrace; |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
55 symbolFunctions['['] = &openBracket; |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
56 symbolFunctions[']'] = &closeBracket; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
57 symbolFunctions[';'] = &seperator; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
58 symbolFunctions[':'] = : |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
59 symbolFunctions['.'] = ˙ |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
60 symbolFunctions[','] = , |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
61 symbolFunctions['='] = &eq; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
62 symbolFunctions['!'] = ≠ |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
63 symbolFunctions['<'] = ≤ |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
64 symbolFunctions['>'] = ≥ |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
65 symbolFunctions['+'] = + |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
66 symbolFunctions['-'] = − |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
67 symbolFunctions['*'] = ☆ |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
68 symbolFunctions['/'] = &slash; |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
69 symbolFunctions['%'] = &percent; |
176
dc9bf56b7ace
Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents:
154
diff
changeset
|
70 symbolFunctions['&'] = ∧ |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
71 symbolFunctions['"'] = &string; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
72 symbolFunctions['`'] = &string; |
1 | 73 } |
74 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
75 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
76 Get the next token from the source. This method will move the |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
77 internal position forward to the next Token. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
78 |
43
a712c530b7cc
A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents:
42
diff
changeset
|
79 return: A Token - Token.type is TokType.EOF if there is |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
80 no more tokens in the file. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
81 */ |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
82 Token next() |
1 | 83 { |
189
75d0544ddc45
Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents:
186
diff
changeset
|
84 Token res; |
1 | 85 switch (getNextChar) |
86 { | |
87 case CharType.EOF: | |
189
75d0544ddc45
Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents:
186
diff
changeset
|
88 return Token(Tok.EOF, last.location, 0); |
1 | 89 |
90 case CharType.Whitespace: | |
91 position += 1; | |
189
75d0544ddc45
Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents:
186
diff
changeset
|
92 res = this.next; |
75d0544ddc45
Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents:
186
diff
changeset
|
93 break; |
1 | 94 |
95 case CharType.Symbol: | |
189
75d0544ddc45
Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents:
186
diff
changeset
|
96 res = lexSymbol; |
75d0544ddc45
Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents:
186
diff
changeset
|
97 break; |
1 | 98 |
99 case CharType.Letter: | |
189
75d0544ddc45
Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents:
186
diff
changeset
|
100 res = lexLetter; |
75d0544ddc45
Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents:
186
diff
changeset
|
101 break; |
1 | 102 |
103 case CharType.Number: | |
189
75d0544ddc45
Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents:
186
diff
changeset
|
104 res = lexNumber; |
75d0544ddc45
Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents:
186
diff
changeset
|
105 break; |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
106 case CharType.Other: |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
107 messages.report(UnexpectedTok, Loc(position)).fatal(ExitLevel.Lexer); |
1 | 108 } |
189
75d0544ddc45
Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents:
186
diff
changeset
|
109 if (res.type != Tok.EOF) |
75d0544ddc45
Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents:
186
diff
changeset
|
110 last = res; |
75d0544ddc45
Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents:
186
diff
changeset
|
111 return res; |
1 | 112 } |
113 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
114 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
115 Get the next token from the source. This method will NOT move the |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
116 internal position forward, and thereby having no side-effects. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
117 |
43
a712c530b7cc
A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents:
42
diff
changeset
|
118 return: A Token - Token.type is TokType.EOF if there is |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
119 no more tokens in the file. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
120 */ |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
121 Token peek(int skip = 0) |
1 | 122 { |
123 int oldPosition = this.position; | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
124 while (skip-- > 0) |
1 | 125 this.next; |
126 Token t = this.next; | |
127 this.position = oldPosition; | |
128 return t; | |
129 } | |
130 | |
189
75d0544ddc45
Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents:
186
diff
changeset
|
131 Token last; |
1 | 132 private: |
189
75d0544ddc45
Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents:
186
diff
changeset
|
133 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
134 Token eq() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
135 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
136 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
137 return Token(Tok.Eq, Loc(position++ - 1), 2); |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
138 return Token(Tok.Assign, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
139 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
140 Token openBrace() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
141 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
142 return Token(Tok.OpenBrace, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
143 } |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
144 Token closeBrace() |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
145 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
146 return Token(Tok.CloseBrace, Loc(position - 1), 1); |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
147 } |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
148 Token openParentheses() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
149 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
150 return Token(Tok.OpenParentheses, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
151 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
152 Token closeParentheses() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
153 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
154 return Token(Tok.CloseParentheses, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
155 } |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
156 Token openBracket() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
157 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
158 return Token(Tok.OpenBracket, Loc(position - 1), 1); |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
159 } |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
160 Token closeBracket() |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
161 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
162 return Token(Tok.CloseBracket, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
163 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
164 Token seperator() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
165 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
166 return Token(Tok.Seperator, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
167 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
168 Token colon() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
169 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
170 return Token(Tok.Colon, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
171 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
172 Token dot() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
173 { |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
174 int pos = 0; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
175 while(getNextChar(0) == CharType.Number || |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
176 this.source[position + pos + 1] == '_') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
177 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
178 if(getNextChar(0) == CharType.Number) |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
179 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
180 position--; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
181 return lexNumber(); |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
182 } |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
183 pos++; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
184 } |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
185 return Token(Tok.Dot, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
186 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
187 Token comma() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
188 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
189 return Token(Tok.Comma, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
190 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
191 Token ne() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
192 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
193 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
194 return Token(Tok.Ne, Loc(position++ - 1), 2); |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
195 return Token(Tok.Not, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
196 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
197 Token le() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
198 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
199 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
200 return Token(Tok.Le, Loc(position++ - 1), 2); |
123 | 201 if(source[position] == '<') |
202 return Token(Tok.LeftShift, Loc(position++ - 1), 2); | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
203 return Token(Tok.Lt, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
204 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
205 Token ge() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
206 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
207 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
208 return Token(Tok.Ge, Loc(position++ - 1), 2); |
123 | 209 if(source[position] == '>') |
210 if(source[position+1] == '>') | |
211 { | |
212 position += 2; | |
213 return Token(Tok.UnsignedRightShift, Loc(position - 1), 3); | |
214 } | |
215 else | |
216 return Token(Tok.RightShift, Loc(position++ - 1), 2); | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
217 return Token(Tok.Gt, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
218 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
219 Token plus() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
220 { |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
221 if(source[position] == '=') |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
222 return Token(Tok.PlusAssign, Loc(position++ - 1), 2); |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
223 return Token(Tok.Plus, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
224 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
225 Token minus() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
226 { |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
227 if(source[position] == '=') |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
228 return Token(Tok.MinusAssign, Loc(position++ - 1), 2); |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
229 return Token(Tok.Minus, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
230 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
231 Token star() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
232 { |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
233 if(source[position] == '=') |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
234 return Token(Tok.StarAssign, Loc(position++ - 1), 2); |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
235 return Token(Tok.Star, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
236 } |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
237 Token slash() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
238 { |
186
e1e170c2cd44
Fixed a error in the test program.
Anders Johnsen <skabet@gmail.com>
parents:
176
diff
changeset
|
239 int p = position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
240 switch(source[position]) |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
241 { |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
242 case '=': |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
243 return Token(Tok.SlashAssign, Loc(position++ - 1), 2); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
244 case '/': |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
245 while(getNextChar != CharType.EOF) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
246 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
247 if(source[position++] == '\n') |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
248 return this.next; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
249 } |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
250 return Token(Tok.EOF, Loc(position), 0); |
1 | 251 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
252 case '*': |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
253 position += 2; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
254 while(getNextChar != CharType.EOF) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
255 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
256 ++position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
257 if(source[position-2] == '*') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
258 if(source[position-1] == '/') |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
259 { |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
260 return this.next; |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
261 } |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
262 } |
186
e1e170c2cd44
Fixed a error in the test program.
Anders Johnsen <skabet@gmail.com>
parents:
176
diff
changeset
|
263 messages.report(UnexpectedEOFBlock,Loc(p)).fatal(ExitLevel.Lexer); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
264 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
265 case '+': |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
266 position += 2; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
267 int nesting = 1; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
268 while(getNextChar != CharType.EOF) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
269 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
270 ++position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
271 if(source[position-2] == '+') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
272 if(source[position-1] == '/') |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
273 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
274 position++; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
275 nesting--; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
276 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
277 |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
278 if(source[position-2] == '/') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
279 if(source[position-1] == '+') |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
280 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
281 nesting++; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
282 position++; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
283 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
284 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
285 if(nesting == 0) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
286 return this.next; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
287 } |
136 | 288 messages.report( |
289 UnexpectedEOFBlock, | |
186
e1e170c2cd44
Fixed a error in the test program.
Anders Johnsen <skabet@gmail.com>
parents:
176
diff
changeset
|
290 Loc(p)).fatal(ExitLevel.Lexer); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
291 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
292 default: |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
293 return Token(Tok.Slash, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
294 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
295 } |
176
dc9bf56b7ace
Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents:
154
diff
changeset
|
296 Token and() |
dc9bf56b7ace
Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents:
154
diff
changeset
|
297 { |
dc9bf56b7ace
Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents:
154
diff
changeset
|
298 return Token(Tok.And, Loc(position - 1), 1); |
dc9bf56b7ace
Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents:
154
diff
changeset
|
299 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
300 Token percent() |
62
78a6808b2e0f
Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents:
43
diff
changeset
|
301 { |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
302 if(source[position] == '=') |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
303 return Token(Tok.PercentAssign, Loc(position++ - 1), 2); |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
304 return Token(Tok.Percent, Loc(position - 1), 1); |
62
78a6808b2e0f
Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents:
43
diff
changeset
|
305 } |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
306 |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
307 Token string() |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
308 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
309 --position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
310 int start = position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
311 if(getNextChar() == CharType.Letter) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
312 position++; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
313 char end = '`'; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
314 switch(source[position]) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
315 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
316 case '"': |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
317 if(position > 0) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
318 if(source[position-1] == 'r') |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
319 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
320 end = '"'; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
321 goto string_wys; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
322 } |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
323 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
324 while(getNextChar != CharType.EOF) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
325 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
326 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
327 if (source[position-1] == '"' ) |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
328 { |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
329 if(getNextChar != CharType.EOF) |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
330 if (source[position] == 'c' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
331 source[position] == 'w' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
332 source[position] == 'd') |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
333 position++; |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
334 |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
335 return Token(Tok.String, Loc(start), position - start); |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
336 } |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
337 else if (source[position-1] == '\\') |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
338 position++; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
339 } |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
340 break; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
341 case '`': |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
342 string_wys: |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
343 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
344 while(getNextChar != CharType.EOF) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
345 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
346 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
347 if (source[position-1] == end ) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
348 return Token(Tok.String, Loc(start), position - start); |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
349 } |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
350 break; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
351 } |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
352 messages.report(UnexpectedEOFBlock, Loc(position)).fatal(ExitLevel.Lexer); |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
353 } |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
354 |
1 | 355 Token lexNumber () |
356 { | |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
357 bool sign; |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
358 |
1 | 359 int i = 0; |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
360 |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
361 |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
362 bool end = false; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
363 while(!end) |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
364 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
365 switch(getNextChar(i)) |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
366 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
367 case CharType.Number: |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
368 break; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
369 case CharType.Symbol: |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
370 if(this.source[position+i] == '.') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
371 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
372 break; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
373 } |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
374 if (this.source[position+i] == '+' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
375 this.source[position+i] == '-') |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
376 { |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
377 if (source[position+i-1] == 'e' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
378 source[position+i-1] == 'E') |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
379 break; |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
380 } |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
381 end = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
382 continue; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
383 case CharType.Letter: |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
384 if(this.source[position+i] == '_') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
385 break; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
386 if (this.source[position+i] == 'e' || |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
387 this.source[position+i] == 'E') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
388 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
389 break; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
390 } |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
391 end = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
392 continue; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
393 |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
394 default: |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
395 end = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
396 continue; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
397 } |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
398 i++; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
399 } |
1 | 400 |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
401 while(source[position+i] == 'u' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
402 source[position+i] == 'U' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
403 source[position+i] == 'L') |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
404 i += 1; |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
405 |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
406 |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
407 |
1 | 408 position += i; |
409 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
410 return Token(Tok.Integer, Loc(position - i), i); |
1 | 411 } |
412 | |
413 Token lexSymbol () | |
414 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
415 Token t = symbolFunctions[source[position++]](); |
1 | 416 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
417 return t; |
1 | 418 } |
419 | |
420 Token lexLetter () | |
421 { | |
422 int i = 0; | |
423 bool hasNumber = false; | |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
424 if (source[position+1] == '"' || |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
425 source[position+1] == '`') |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
426 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
427 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
428 return string; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
429 } |
1 | 430 while (getNextChar(++i) == CharType.Letter || |
431 getNextChar(i) == CharType.Number) | |
432 { | |
433 if (getNextChar(i) == CharType.Number) | |
434 { | |
435 hasNumber = true; | |
436 } | |
437 } | |
438 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
439 Token t = Token(Tok.Identifier, Loc(), i); |
1 | 440 |
441 if (!hasNumber) | |
442 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
443 char[] str = source[position .. position + i]; |
1 | 444 if(str in keywords) |
445 t.type = keywords[str]; | |
446 } | |
447 | |
448 position += i; | |
449 | |
450 return t; | |
451 } | |
452 | |
453 CharType getNextChar(int offset = 0) | |
454 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
455 if (position + offset >= this.source.length) |
1 | 456 return CharType.EOF; |
457 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
458 char current = source[position + offset]; |
1 | 459 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
460 CharType c = charTable[current]; |
1 | 461 |
43
a712c530b7cc
A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents:
42
diff
changeset
|
462 if(c == CharType.INVALID) |
92
771ac63898e2
A few better parser errors plus renaming most of the sema classes to match that they do now. Some have changes a lot.
Anders Johnsen <skabet@gmail.com>
parents:
91
diff
changeset
|
463 messages.report(InvalidSymbol, Loc()) |
771ac63898e2
A few better parser errors plus renaming most of the sema classes to match that they do now. Some have changes a lot.
Anders Johnsen <skabet@gmail.com>
parents:
91
diff
changeset
|
464 .arg(Integer.toString(cast(int)current)) |
771ac63898e2
A few better parser errors plus renaming most of the sema classes to match that they do now. Some have changes a lot.
Anders Johnsen <skabet@gmail.com>
parents:
91
diff
changeset
|
465 .fatal(ExitLevel.Lexer); |
1 | 466 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
467 return c; |
1 | 468 |
469 } | |
470 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
471 private final SourceLocation Loc(int pos = -1) |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
472 { |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
473 if (pos < 0) |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
474 return start_loc + position; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
475 return start_loc + pos; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
476 } |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
477 |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
478 SourceManager sm; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
479 SourceLocation start_loc; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
480 int position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
481 char[] source; |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
482 MessageHandler messages; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
483 CharType[] charTable; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
484 Token delegate()[] symbolFunctions; |
1 | 485 } |
486 | |
487 enum CharType : ubyte | |
488 { | |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
489 INVALID, |
1 | 490 Letter, |
491 Number, | |
492 Symbol, | |
493 Whitespace, | |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
494 Other, |
1 | 495 |
496 EOF | |
497 } | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
498 |