Mercurial > projects > dang
annotate lexer/Lexer.d @ 186:e1e170c2cd44
Fixed a error in the test program.
author | Anders Johnsen <skabet@gmail.com> |
---|---|
date | Fri, 25 Jul 2008 12:50:09 +0200 |
parents | dc9bf56b7ace |
children | 75d0544ddc45 |
rev | line source |
---|---|
1 | 1 module lexer.Lexer; |
2 | |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
3 import basic.Message, |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
4 basic.SourceManager; |
1 | 5 |
6 import lexer.Token, | |
7 lexer.Keyword; | |
8 | |
9 import tango.io.Stdout; | |
10 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
11 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
12 The Lexer class will supply you with methods to tokenize a D file. Supply the |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
13 Lexer with a DataSource and you can 'peek' and 'next' Tokens from the file. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
14 |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
15 For more info about Tokens, look up the lexer.Token module. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
16 */ |
1 | 17 class Lexer |
18 { | |
19 public: | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
20 |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
21 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
22 Create a new Lexer. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
23 */ |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
24 this(SourceLocation start, SourceManager src_mgr, MessageHandler messages) |
1 | 25 { |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
26 this.messages = messages; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
27 sm = src_mgr; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
28 start_loc = start; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
29 position = 0; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
30 source = sm.getRawData(start_loc); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
31 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
32 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
33 charTable.length = 256; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
34 foreach (c; "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
35 charTable[c] = CharType.Letter; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
36 |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
37 foreach (c; "0123456789") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
38 charTable[c] = CharType.Number; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
39 |
176
dc9bf56b7ace
Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents:
154
diff
changeset
|
40 foreach (c; "(){}[];:.,=!<>+-*/%&\"`") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
41 charTable[c] = CharType.Symbol; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
42 |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
43 foreach (c; " \n") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
44 charTable[c] = CharType.Whitespace; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
45 |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
46 foreach (c; "'\\") |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
47 charTable[c] = CharType.Other; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
48 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
49 symbolFunctions.length = 256; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
50 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
51 symbolFunctions['('] = &openParentheses; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
52 symbolFunctions[')'] = &closeParentheses; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
53 symbolFunctions['{'] = &openBrace; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
54 symbolFunctions['}'] = &closeBrace; |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
55 symbolFunctions['['] = &openBracket; |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
56 symbolFunctions[']'] = &closeBracket; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
57 symbolFunctions[';'] = &seperator; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
58 symbolFunctions[':'] = : |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
59 symbolFunctions['.'] = ˙ |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
60 symbolFunctions[','] = , |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
61 symbolFunctions['='] = &eq; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
62 symbolFunctions['!'] = ≠ |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
63 symbolFunctions['<'] = ≤ |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
64 symbolFunctions['>'] = ≥ |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
65 symbolFunctions['+'] = + |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
66 symbolFunctions['-'] = − |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
67 symbolFunctions['*'] = ☆ |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
68 symbolFunctions['/'] = &slash; |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
69 symbolFunctions['%'] = &percent; |
176
dc9bf56b7ace
Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents:
154
diff
changeset
|
70 symbolFunctions['&'] = ∧ |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
71 symbolFunctions['"'] = &string; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
72 symbolFunctions['`'] = &string; |
1 | 73 } |
74 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
75 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
76 Get the next token from the source. This method will move the |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
77 internal position forward to the next Token. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
78 |
43
a712c530b7cc
A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents:
42
diff
changeset
|
79 return: A Token - Token.type is TokType.EOF if there is |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
80 no more tokens in the file. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
81 */ |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
82 Token next() |
1 | 83 { |
84 switch (getNextChar) | |
85 { | |
86 case CharType.EOF: | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
87 SLoc loc; |
154
0ea5d2f3e96b
Parsing "this" as constructor. Also removed regex from the test run program(seg fault - dmd???)
Anders Johnsen <skabet@gmail.com>
parents:
136
diff
changeset
|
88 return Token(Tok.EOF, loc+1, 0); |
1 | 89 |
90 case CharType.Whitespace: | |
91 position += 1; | |
92 return this.next; | |
93 | |
94 case CharType.Symbol: | |
95 return lexSymbol; | |
96 | |
97 case CharType.Letter: | |
98 return lexLetter; | |
99 | |
100 case CharType.Number: | |
101 return lexNumber; | |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
102 case CharType.Other: |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
103 messages.report(UnexpectedTok, Loc(position)).fatal(ExitLevel.Lexer); |
1 | 104 } |
105 } | |
106 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
107 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
108 Get the next token from the source. This method will NOT move the |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
109 internal position forward, and thereby having no side-effects. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
110 |
43
a712c530b7cc
A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents:
42
diff
changeset
|
111 return: A Token - Token.type is TokType.EOF if there is |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
112 no more tokens in the file. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
113 */ |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
114 Token peek(int skip = 0) |
1 | 115 { |
116 int oldPosition = this.position; | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
117 while (skip-- > 0) |
1 | 118 this.next; |
119 Token t = this.next; | |
120 this.position = oldPosition; | |
121 return t; | |
122 } | |
123 | |
124 private: | |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
125 Token eq() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
126 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
127 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
128 return Token(Tok.Eq, Loc(position++ - 1), 2); |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
129 return Token(Tok.Assign, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
130 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
131 Token openBrace() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
132 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
133 return Token(Tok.OpenBrace, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
134 } |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
135 Token closeBrace() |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
136 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
137 return Token(Tok.CloseBrace, Loc(position - 1), 1); |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
138 } |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
139 Token openParentheses() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
140 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
141 return Token(Tok.OpenParentheses, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
142 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
143 Token closeParentheses() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
144 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
145 return Token(Tok.CloseParentheses, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
146 } |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
147 Token openBracket() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
148 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
149 return Token(Tok.OpenBracket, Loc(position - 1), 1); |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
150 } |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
151 Token closeBracket() |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
152 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
153 return Token(Tok.CloseBracket, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
154 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
155 Token seperator() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
156 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
157 return Token(Tok.Seperator, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
158 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
159 Token colon() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
160 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
161 return Token(Tok.Colon, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
162 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
163 Token dot() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
164 { |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
165 int pos = 0; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
166 while(getNextChar(0) == CharType.Number || |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
167 this.source[position + pos + 1] == '_') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
168 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
169 if(getNextChar(0) == CharType.Number) |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
170 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
171 position--; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
172 return lexNumber(); |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
173 } |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
174 pos++; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
175 } |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
176 return Token(Tok.Dot, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
177 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
178 Token comma() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
179 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
180 return Token(Tok.Comma, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
181 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
182 Token ne() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
183 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
184 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
185 return Token(Tok.Ne, Loc(position++ - 1), 2); |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
186 return Token(Tok.Not, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
187 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
188 Token le() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
189 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
190 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
191 return Token(Tok.Le, Loc(position++ - 1), 2); |
123 | 192 if(source[position] == '<') |
193 return Token(Tok.LeftShift, Loc(position++ - 1), 2); | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
194 return Token(Tok.Lt, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
195 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
196 Token ge() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
197 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
198 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
199 return Token(Tok.Ge, Loc(position++ - 1), 2); |
123 | 200 if(source[position] == '>') |
201 if(source[position+1] == '>') | |
202 { | |
203 position += 2; | |
204 return Token(Tok.UnsignedRightShift, Loc(position - 1), 3); | |
205 } | |
206 else | |
207 return Token(Tok.RightShift, Loc(position++ - 1), 2); | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
208 return Token(Tok.Gt, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
209 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
210 Token plus() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
211 { |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
212 if(source[position] == '=') |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
213 return Token(Tok.PlusAssign, Loc(position++ - 1), 2); |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
214 return Token(Tok.Plus, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
215 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
216 Token minus() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
217 { |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
218 if(source[position] == '=') |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
219 return Token(Tok.MinusAssign, Loc(position++ - 1), 2); |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
220 return Token(Tok.Minus, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
221 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
222 Token star() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
223 { |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
224 if(source[position] == '=') |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
225 return Token(Tok.StarAssign, Loc(position++ - 1), 2); |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
226 return Token(Tok.Star, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
227 } |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
228 Token slash() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
229 { |
186
e1e170c2cd44
Fixed a error in the test program.
Anders Johnsen <skabet@gmail.com>
parents:
176
diff
changeset
|
230 int p = position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
231 switch(source[position]) |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
232 { |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
233 case '=': |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
234 return Token(Tok.SlashAssign, Loc(position++ - 1), 2); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
235 case '/': |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
236 while(getNextChar != CharType.EOF) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
237 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
238 if(source[position++] == '\n') |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
239 return this.next; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
240 } |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
241 return Token(Tok.EOF, Loc(position), 0); |
1 | 242 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
243 case '*': |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
244 position += 2; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
245 while(getNextChar != CharType.EOF) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
246 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
247 ++position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
248 if(source[position-2] == '*') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
249 if(source[position-1] == '/') |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
250 { |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
251 return this.next; |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
252 } |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
253 } |
186
e1e170c2cd44
Fixed a error in the test program.
Anders Johnsen <skabet@gmail.com>
parents:
176
diff
changeset
|
254 messages.report(UnexpectedEOFBlock,Loc(p)).fatal(ExitLevel.Lexer); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
255 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
256 case '+': |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
257 position += 2; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
258 int nesting = 1; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
259 while(getNextChar != CharType.EOF) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
260 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
261 ++position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
262 if(source[position-2] == '+') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
263 if(source[position-1] == '/') |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
264 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
265 position++; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
266 nesting--; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
267 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
268 |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
269 if(source[position-2] == '/') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
270 if(source[position-1] == '+') |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
271 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
272 nesting++; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
273 position++; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
274 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
275 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
276 if(nesting == 0) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
277 return this.next; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
278 } |
136 | 279 messages.report( |
280 UnexpectedEOFBlock, | |
186
e1e170c2cd44
Fixed a error in the test program.
Anders Johnsen <skabet@gmail.com>
parents:
176
diff
changeset
|
281 Loc(p)).fatal(ExitLevel.Lexer); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
282 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
283 default: |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
284 return Token(Tok.Slash, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
285 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
286 } |
176
dc9bf56b7ace
Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents:
154
diff
changeset
|
287 Token and() |
dc9bf56b7ace
Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents:
154
diff
changeset
|
288 { |
dc9bf56b7ace
Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents:
154
diff
changeset
|
289 return Token(Tok.And, Loc(position - 1), 1); |
dc9bf56b7ace
Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents:
154
diff
changeset
|
290 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
291 Token percent() |
62
78a6808b2e0f
Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents:
43
diff
changeset
|
292 { |
126
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
293 if(source[position] == '=') |
c3b24e7e8cf8
Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents:
123
diff
changeset
|
294 return Token(Tok.PercentAssign, Loc(position++ - 1), 2); |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
295 return Token(Tok.Percent, Loc(position - 1), 1); |
62
78a6808b2e0f
Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents:
43
diff
changeset
|
296 } |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
297 |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
298 Token string() |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
299 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
300 --position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
301 int start = position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
302 if(getNextChar() == CharType.Letter) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
303 position++; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
304 char end = '`'; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
305 switch(source[position]) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
306 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
307 case '"': |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
308 if(position > 0) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
309 if(source[position-1] == 'r') |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
310 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
311 end = '"'; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
312 goto string_wys; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
313 } |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
314 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
315 while(getNextChar != CharType.EOF) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
316 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
317 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
318 if (source[position-1] == '"' ) |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
319 { |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
320 if(getNextChar != CharType.EOF) |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
321 if (source[position] == 'c' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
322 source[position] == 'w' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
323 source[position] == 'd') |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
324 position++; |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
325 |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
326 return Token(Tok.String, Loc(start), position - start); |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
327 } |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
328 else if (source[position-1] == '\\') |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
329 position++; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
330 } |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
331 break; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
332 case '`': |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
333 string_wys: |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
334 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
335 while(getNextChar != CharType.EOF) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
336 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
337 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
338 if (source[position-1] == end ) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
339 return Token(Tok.String, Loc(start), position - start); |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
340 } |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
341 break; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
342 } |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
343 messages.report(UnexpectedEOFBlock, Loc(position)).fatal(ExitLevel.Lexer); |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
344 } |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
345 |
1 | 346 Token lexNumber () |
347 { | |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
348 bool sign; |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
349 |
1 | 350 int i = 0; |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
351 |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
352 |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
353 bool end = false; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
354 while(!end) |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
355 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
356 switch(getNextChar(i)) |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
357 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
358 case CharType.Number: |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
359 break; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
360 case CharType.Symbol: |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
361 if(this.source[position+i] == '.') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
362 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
363 break; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
364 } |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
365 if (this.source[position+i] == '+' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
366 this.source[position+i] == '-') |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
367 { |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
368 if (source[position+i-1] == 'e' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
369 source[position+i-1] == 'E') |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
370 break; |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
371 } |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
372 end = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
373 continue; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
374 case CharType.Letter: |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
375 if(this.source[position+i] == '_') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
376 break; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
377 if (this.source[position+i] == 'e' || |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
378 this.source[position+i] == 'E') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
379 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
380 break; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
381 } |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
382 end = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
383 continue; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
384 |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
385 default: |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
386 end = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
387 continue; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
388 } |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
389 i++; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
390 } |
1 | 391 |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
392 while(source[position+i] == 'u' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
393 source[position+i] == 'U' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
394 source[position+i] == 'L') |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
395 i += 1; |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
396 |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
397 |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
398 |
1 | 399 position += i; |
400 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
401 return Token(Tok.Integer, Loc(position - i), i); |
1 | 402 } |
403 | |
404 Token lexSymbol () | |
405 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
406 Token t = symbolFunctions[source[position++]](); |
1 | 407 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
408 return t; |
1 | 409 } |
410 | |
411 Token lexLetter () | |
412 { | |
413 int i = 0; | |
414 bool hasNumber = false; | |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
415 if (source[position+1] == '"' || |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
416 source[position+1] == '`') |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
417 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
418 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
419 return string; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
420 } |
1 | 421 while (getNextChar(++i) == CharType.Letter || |
422 getNextChar(i) == CharType.Number) | |
423 { | |
424 if (getNextChar(i) == CharType.Number) | |
425 { | |
426 hasNumber = true; | |
427 } | |
428 } | |
429 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
430 Token t = Token(Tok.Identifier, Loc(), i); |
1 | 431 |
432 if (!hasNumber) | |
433 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
434 char[] str = source[position .. position + i]; |
1 | 435 if(str in keywords) |
436 t.type = keywords[str]; | |
437 } | |
438 | |
439 position += i; | |
440 | |
441 return t; | |
442 } | |
443 | |
444 CharType getNextChar(int offset = 0) | |
445 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
446 if (position + offset >= this.source.length) |
1 | 447 return CharType.EOF; |
448 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
449 char current = source[position + offset]; |
1 | 450 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
451 CharType c = charTable[current]; |
1 | 452 |
43
a712c530b7cc
A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents:
42
diff
changeset
|
453 if(c == CharType.INVALID) |
92
771ac63898e2
A few better parser errors plus renaming most of the sema classes to match that they do now. Some have changes a lot.
Anders Johnsen <skabet@gmail.com>
parents:
91
diff
changeset
|
454 messages.report(InvalidSymbol, Loc()) |
771ac63898e2
A few better parser errors plus renaming most of the sema classes to match that they do now. Some have changes a lot.
Anders Johnsen <skabet@gmail.com>
parents:
91
diff
changeset
|
455 .arg(Integer.toString(cast(int)current)) |
771ac63898e2
A few better parser errors plus renaming most of the sema classes to match that they do now. Some have changes a lot.
Anders Johnsen <skabet@gmail.com>
parents:
91
diff
changeset
|
456 .fatal(ExitLevel.Lexer); |
1 | 457 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
458 return c; |
1 | 459 |
460 } | |
461 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
462 private final SourceLocation Loc(int pos = -1) |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
463 { |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
464 if (pos < 0) |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
465 return start_loc + position; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
466 return start_loc + pos; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
467 } |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
468 |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
469 SourceManager sm; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
470 SourceLocation start_loc; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
471 int position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
472 char[] source; |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
473 MessageHandler messages; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
474 CharType[] charTable; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
475 Token delegate()[] symbolFunctions; |
1 | 476 } |
477 | |
478 enum CharType : ubyte | |
479 { | |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
480 INVALID, |
1 | 481 Letter, |
482 Number, | |
483 Symbol, | |
484 Whitespace, | |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
485 Other, |
1 | 486 |
487 EOF | |
488 } | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
489 |