Mercurial > projects > dang
annotate lexer/Lexer.d @ 89:a49bb982a7b0 new_gen
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
author | Anders Johnsen <skabet@gmail.com> |
---|---|
date | Sun, 04 May 2008 20:27:01 +0200 |
parents | eb5b2c719a39 |
children | 1a24e61eb104 |
rev | line source |
---|---|
1 | 1 module lexer.Lexer; |
2 | |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
3 import basic.Message, |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
4 basic.SourceManager; |
1 | 5 |
6 import lexer.Token, | |
7 lexer.Keyword; | |
8 | |
9 import tango.io.Stdout; | |
10 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
11 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
12 The Lexer class will supply you with methods to tokenize a D file. Supply the |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
13 Lexer with a DataSource and you can 'peek' and 'next' Tokens from the file. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
14 |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
15 For more info about Tokens, look up the lexer.Token module. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
16 */ |
1 | 17 class Lexer |
18 { | |
19 public: | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
20 |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
21 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
22 Create a new Lexer. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
23 */ |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
24 |
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
25 |
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
26 this(SourceLocation start, SourceManager src_mgr, MessageHandler messages) |
1 | 27 { |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
28 this.messages = messages; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
29 sm = src_mgr; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
30 start_loc = start; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
31 position = 0; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
32 source = sm.getRawData(start_loc); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
33 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
34 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
35 charTable.length = 256; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
36 foreach (c; "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
37 charTable[c] = CharType.Letter; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
38 |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
39 foreach (c; "0123456789") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
40 charTable[c] = CharType.Number; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
41 |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
42 foreach (c; "(){}[];:.,=!<>+-*/%") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
43 charTable[c] = CharType.Symbol; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
44 |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
45 foreach (c; " \n") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
46 charTable[c] = CharType.Whitespace; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
47 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
48 symbolFunctions.length = 256; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
49 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
50 symbolFunctions['('] = &openParentheses; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
51 symbolFunctions[')'] = &closeParentheses; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
52 symbolFunctions['{'] = &openBrace; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
53 symbolFunctions['}'] = &closeBrace; |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
54 symbolFunctions['['] = &openBracket; |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
55 symbolFunctions[']'] = &closeBracket; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
56 symbolFunctions[';'] = &seperator; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
57 symbolFunctions[':'] = : |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
58 symbolFunctions['.'] = ˙ |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
59 symbolFunctions[','] = , |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
60 symbolFunctions['='] = &eq; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
61 symbolFunctions['!'] = ≠ |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
62 symbolFunctions['<'] = ≤ |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
63 symbolFunctions['>'] = ≥ |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
64 symbolFunctions['+'] = + |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
65 symbolFunctions['-'] = − |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
66 symbolFunctions['*'] = ☆ |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
67 symbolFunctions['/'] = &slash; |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
68 symbolFunctions['%'] = &percent; |
1 | 69 } |
70 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
71 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
72 Get the next token from the source. This method will move the |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
73 internal position forward to the next Token. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
74 |
43
a712c530b7cc
A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents:
42
diff
changeset
|
75 return: A Token - Token.type is TokType.EOF if there is |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
76 no more tokens in the file. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
77 */ |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
78 Token next() |
1 | 79 { |
80 switch (getNextChar) | |
81 { | |
82 case CharType.EOF: | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
83 SLoc loc; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
84 return Token(Tok.EOF, loc, 0); |
1 | 85 |
86 case CharType.Whitespace: | |
87 position += 1; | |
88 return this.next; | |
89 | |
90 case CharType.Symbol: | |
91 return lexSymbol; | |
92 | |
93 case CharType.Letter: | |
94 return lexLetter; | |
95 | |
96 case CharType.Number: | |
97 return lexNumber; | |
98 } | |
99 } | |
100 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
101 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
102 Get the next token from the source. This method will NOT move the |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
103 internal position forward, and thereby having no side-effects. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
104 |
43
a712c530b7cc
A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents:
42
diff
changeset
|
105 return: A Token - Token.type is TokType.EOF if there is |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
106 no more tokens in the file. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
107 */ |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
108 Token peek(int skip = 0) |
1 | 109 { |
110 int oldPosition = this.position; | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
111 while (skip-- > 0) |
1 | 112 this.next; |
113 Token t = this.next; | |
114 this.position = oldPosition; | |
115 return t; | |
116 } | |
117 | |
118 private: | |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
119 Token eq() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
120 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
121 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
122 return Token(Tok.Eq, Loc(position++ - 1), 2); |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
123 return Token(Tok.Assign, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
124 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
125 Token openBrace() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
126 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
127 return Token(Tok.OpenBrace, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
128 } |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
129 Token closeBrace() |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
130 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
131 return Token(Tok.CloseBrace, Loc(position - 1), 1); |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
132 } |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
133 Token openParentheses() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
134 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
135 return Token(Tok.OpenParentheses, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
136 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
137 Token closeParentheses() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
138 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
139 return Token(Tok.CloseParentheses, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
140 } |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
141 Token openBracket() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
142 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
143 return Token(Tok.OpenBracket, Loc(position - 1), 1); |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
144 } |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
145 Token closeBracket() |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
146 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
147 return Token(Tok.CloseBracket, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
148 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
149 Token seperator() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
150 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
151 return Token(Tok.Seperator, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
152 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
153 Token colon() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
154 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
155 return Token(Tok.Colon, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
156 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
157 Token dot() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
158 { |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
159 int pos = 0; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
160 while(getNextChar(0) == CharType.Number || |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
161 this.source[position + pos + 1] == '_') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
162 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
163 if(getNextChar(0) == CharType.Number) |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
164 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
165 position--; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
166 return lexNumber(); |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
167 } |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
168 pos++; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
169 } |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
170 return Token(Tok.Dot, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
171 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
172 Token comma() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
173 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
174 return Token(Tok.Comma, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
175 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
176 Token ne() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
177 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
178 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
179 return Token(Tok.Ne, Loc(position++ - 1), 2); |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
180 return Token(Tok.Not, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
181 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
182 Token le() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
183 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
184 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
185 return Token(Tok.Le, Loc(position++ - 1), 2); |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
186 return Token(Tok.Lt, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
187 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
188 Token ge() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
189 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
190 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
191 return Token(Tok.Ge, Loc(position++ - 1), 2); |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
192 return Token(Tok.Gt, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
193 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
194 Token plus() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
195 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
196 return Token(Tok.Plus, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
197 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
198 Token minus() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
199 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
200 return Token(Tok.Minus, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
201 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
202 Token star() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
203 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
204 return Token(Tok.Star, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
205 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
206 Token slash() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
207 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
208 switch(source[position]) |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
209 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
210 case '/': |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
211 while(getNextChar != CharType.EOF) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
212 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
213 if(source[position++] == '\n') |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
214 return this.next; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
215 } |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
216 return Token(Tok.EOF, Loc(position), 0); |
1 | 217 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
218 case '*': |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
219 position += 2; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
220 while(getNextChar != CharType.EOF) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
221 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
222 ++position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
223 if(source[position-2] == '*') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
224 if(source[position-1] == '/') |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
225 return this.next; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
226 } |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
227 messages.report(UnexpectedEOFBlock,Loc(position)); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
228 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
229 case '+': |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
230 position += 2; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
231 int nesting = 1; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
232 while(getNextChar != CharType.EOF) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
233 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
234 ++position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
235 if(source[position-2] == '+') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
236 if(source[position-1] == '/') |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
237 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
238 position++; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
239 nesting--; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
240 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
241 |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
242 if(source[position-2] == '/') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
243 if(source[position-1] == '+') |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
244 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
245 nesting++; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
246 position++; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
247 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
248 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
249 if(nesting == 0) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
250 return this.next; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
251 } |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
252 messages.report(UnexpectedEOFBlock,Loc(position)); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
253 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
254 default: |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
255 return Token(Tok.Slash, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
256 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
257 } |
62
78a6808b2e0f
Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents:
43
diff
changeset
|
258 |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
259 Token percent() |
62
78a6808b2e0f
Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents:
43
diff
changeset
|
260 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
261 return Token(Tok.Percent, Loc(position - 1), 1); |
62
78a6808b2e0f
Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents:
43
diff
changeset
|
262 } |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
263 |
1 | 264 Token lexNumber () |
265 { | |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
266 bool sign = false; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
267 bool dot = false; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
268 bool e = false; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
269 |
1 | 270 int i = 0; |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
271 |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
272 bool end = false; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
273 while(!end) |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
274 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
275 switch(getNextChar(i)) |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
276 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
277 case CharType.Number: |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
278 break; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
279 case CharType.Symbol: |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
280 if(this.source[position+i] == '.') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
281 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
282 if(dot) |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
283 messages.report(OnlyOneDotFloating, Loc(position + i)); |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
284 dot = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
285 break; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
286 } |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
287 end = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
288 continue; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
289 case CharType.Letter: |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
290 if(this.source[position+i] == '_') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
291 break; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
292 if (this.source[position+i] == 'e' || |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
293 this.source[position+i] == 'E') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
294 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
295 if (e) |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
296 messages.report(OnlyOneEFloating, Loc(position + i)); |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
297 e = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
298 break; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
299 } |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
300 end = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
301 continue; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
302 |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
303 default: |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
304 end = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
305 continue; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
306 } |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
307 i++; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
308 } |
1 | 309 |
310 position += i; | |
311 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
312 return Token(Tok.Integer, Loc(position - i), i); |
1 | 313 } |
314 | |
315 Token lexSymbol () | |
316 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
317 Token t = symbolFunctions[source[position++]](); |
1 | 318 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
319 return t; |
1 | 320 } |
321 | |
322 Token lexLetter () | |
323 { | |
324 int i = 0; | |
325 bool hasNumber = false; | |
326 while (getNextChar(++i) == CharType.Letter || | |
327 getNextChar(i) == CharType.Number) | |
328 { | |
329 if (getNextChar(i) == CharType.Number) | |
330 { | |
331 hasNumber = true; | |
332 } | |
333 } | |
334 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
335 Token t = Token(Tok.Identifier, Loc(), i); |
1 | 336 |
337 if (!hasNumber) | |
338 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
339 char[] str = source[position .. position + i]; |
1 | 340 if(str in keywords) |
341 t.type = keywords[str]; | |
342 } | |
343 | |
344 position += i; | |
345 | |
346 return t; | |
347 } | |
348 | |
349 CharType getNextChar(int offset = 0) | |
350 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
351 if (position + offset >= this.source.length) |
1 | 352 return CharType.EOF; |
353 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
354 char current = source[position + offset]; |
1 | 355 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
356 CharType c = charTable[current]; |
1 | 357 |
43
a712c530b7cc
A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents:
42
diff
changeset
|
358 if(c == CharType.INVALID) |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
359 messages.report(InvalidSymbol, SLoc()).arg(current); |
1 | 360 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
361 return c; |
1 | 362 |
363 } | |
364 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
365 private final SourceLocation Loc(int pos = -1) |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
366 { |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
367 if (pos < 0) |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
368 return start_loc + position; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
369 return start_loc + pos; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
370 } |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
371 |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
372 SourceManager sm; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
373 SourceLocation start_loc; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
374 int position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
375 char[] source; |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
376 MessageHandler messages; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
377 CharType[] charTable; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
378 Token delegate()[] symbolFunctions; |
1 | 379 } |
380 | |
381 enum CharType : ubyte | |
382 { | |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
383 INVALID, |
1 | 384 Letter, |
385 Number, | |
386 Symbol, | |
387 Whitespace, | |
388 | |
389 EOF | |
390 } | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
391 |