Mercurial > projects > dang
annotate lexer/Lexer.d @ 123:6a5f745d351c
Parsing <<, >> and >>>.
author | Anders Johnsen <skabet@gmail.com> |
---|---|
date | Sun, 25 May 2008 21:07:48 +0200 |
parents | c658172ca8a0 |
children | c3b24e7e8cf8 |
rev | line source |
---|---|
1 | 1 module lexer.Lexer; |
2 | |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
3 import basic.Message, |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
4 basic.SourceManager; |
1 | 5 |
6 import lexer.Token, | |
7 lexer.Keyword; | |
8 | |
9 import tango.io.Stdout; | |
10 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
11 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
12 The Lexer class will supply you with methods to tokenize a D file. Supply the |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
13 Lexer with a DataSource and you can 'peek' and 'next' Tokens from the file. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
14 |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
15 For more info about Tokens, look up the lexer.Token module. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
16 */ |
1 | 17 class Lexer |
18 { | |
19 public: | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
20 |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
21 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
22 Create a new Lexer. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
23 */ |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
24 this(SourceLocation start, SourceManager src_mgr, MessageHandler messages) |
1 | 25 { |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
26 this.messages = messages; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
27 sm = src_mgr; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
28 start_loc = start; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
29 position = 0; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
30 source = sm.getRawData(start_loc); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
31 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
32 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
33 charTable.length = 256; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
34 foreach (c; "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
35 charTable[c] = CharType.Letter; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
36 |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
37 foreach (c; "0123456789") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
38 charTable[c] = CharType.Number; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
39 |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
40 foreach (c; "(){}[];:.,=!<>+-*/%\"`") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
41 charTable[c] = CharType.Symbol; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
42 |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
43 foreach (c; " \n") |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
44 charTable[c] = CharType.Whitespace; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
45 |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
46 foreach (c; "'\\") |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
47 charTable[c] = CharType.Other; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
48 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
49 symbolFunctions.length = 256; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
50 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
51 symbolFunctions['('] = &openParentheses; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
52 symbolFunctions[')'] = &closeParentheses; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
53 symbolFunctions['{'] = &openBrace; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
54 symbolFunctions['}'] = &closeBrace; |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
55 symbolFunctions['['] = &openBracket; |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
56 symbolFunctions[']'] = &closeBracket; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
57 symbolFunctions[';'] = &seperator; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
58 symbolFunctions[':'] = : |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
59 symbolFunctions['.'] = ˙ |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
60 symbolFunctions[','] = , |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
61 symbolFunctions['='] = &eq; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
62 symbolFunctions['!'] = ≠ |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
63 symbolFunctions['<'] = ≤ |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
64 symbolFunctions['>'] = ≥ |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
65 symbolFunctions['+'] = + |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
66 symbolFunctions['-'] = − |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
67 symbolFunctions['*'] = ☆ |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
68 symbolFunctions['/'] = &slash; |
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
69 symbolFunctions['%'] = &percent; |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
70 symbolFunctions['"'] = &string; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
71 symbolFunctions['`'] = &string; |
1 | 72 } |
73 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
74 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
75 Get the next token from the source. This method will move the |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
76 internal position forward to the next Token. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
77 |
43
a712c530b7cc
A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents:
42
diff
changeset
|
78 return: A Token - Token.type is TokType.EOF if there is |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
79 no more tokens in the file. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
80 */ |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
81 Token next() |
1 | 82 { |
83 switch (getNextChar) | |
84 { | |
85 case CharType.EOF: | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
86 SLoc loc; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
87 return Token(Tok.EOF, loc, 0); |
1 | 88 |
89 case CharType.Whitespace: | |
90 position += 1; | |
91 return this.next; | |
92 | |
93 case CharType.Symbol: | |
94 return lexSymbol; | |
95 | |
96 case CharType.Letter: | |
97 return lexLetter; | |
98 | |
99 case CharType.Number: | |
100 return lexNumber; | |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
101 case CharType.Other: |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
102 messages.report(UnexpectedTok, Loc(position)).fatal(ExitLevel.Lexer); |
1 | 103 } |
104 } | |
105 | |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
106 /** |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
107 Get the next token from the source. This method will NOT move the |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
108 internal position forward, and thereby having no side-effects. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
109 |
43
a712c530b7cc
A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents:
42
diff
changeset
|
110 return: A Token - Token.type is TokType.EOF if there is |
42
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
111 no more tokens in the file. |
4e879f82dd64
Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents:
41
diff
changeset
|
112 */ |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
113 Token peek(int skip = 0) |
1 | 114 { |
115 int oldPosition = this.position; | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
116 while (skip-- > 0) |
1 | 117 this.next; |
118 Token t = this.next; | |
119 this.position = oldPosition; | |
120 return t; | |
121 } | |
122 | |
123 private: | |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
124 Token eq() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
125 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
126 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
127 return Token(Tok.Eq, Loc(position++ - 1), 2); |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
128 return Token(Tok.Assign, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
129 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
130 Token openBrace() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
131 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
132 return Token(Tok.OpenBrace, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
133 } |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
134 Token closeBrace() |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
135 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
136 return Token(Tok.CloseBrace, Loc(position - 1), 1); |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
137 } |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
138 Token openParentheses() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
139 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
140 return Token(Tok.OpenParentheses, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
141 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
142 Token closeParentheses() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
143 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
144 return Token(Tok.CloseParentheses, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
145 } |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
146 Token openBracket() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
147 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
148 return Token(Tok.OpenBracket, Loc(position - 1), 1); |
72
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
149 } |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
150 Token closeBracket() |
628cb46ab13b
First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents:
67
diff
changeset
|
151 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
152 return Token(Tok.CloseBracket, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
153 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
154 Token seperator() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
155 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
156 return Token(Tok.Seperator, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
157 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
158 Token colon() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
159 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
160 return Token(Tok.Colon, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
161 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
162 Token dot() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
163 { |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
164 int pos = 0; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
165 while(getNextChar(0) == CharType.Number || |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
166 this.source[position + pos + 1] == '_') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
167 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
168 if(getNextChar(0) == CharType.Number) |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
169 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
170 position--; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
171 return lexNumber(); |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
172 } |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
173 pos++; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
174 } |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
175 return Token(Tok.Dot, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
176 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
177 Token comma() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
178 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
179 return Token(Tok.Comma, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
180 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
181 Token ne() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
182 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
183 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
184 return Token(Tok.Ne, Loc(position++ - 1), 2); |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
185 return Token(Tok.Not, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
186 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
187 Token le() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
188 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
189 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
190 return Token(Tok.Le, Loc(position++ - 1), 2); |
123 | 191 if(source[position] == '<') |
192 return Token(Tok.LeftShift, Loc(position++ - 1), 2); | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
193 return Token(Tok.Lt, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
194 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
195 Token ge() |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
196 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
197 if(source[position] == '=') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
198 return Token(Tok.Ge, Loc(position++ - 1), 2); |
123 | 199 if(source[position] == '>') |
200 if(source[position+1] == '>') | |
201 { | |
202 position += 2; | |
203 return Token(Tok.UnsignedRightShift, Loc(position - 1), 3); | |
204 } | |
205 else | |
206 return Token(Tok.RightShift, Loc(position++ - 1), 2); | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
207 return Token(Tok.Gt, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
208 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
209 Token plus() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
210 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
211 return Token(Tok.Plus, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
212 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
213 Token minus() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
214 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
215 return Token(Tok.Minus, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
216 } |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
217 Token star() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
218 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
219 return Token(Tok.Star, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
220 } |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
221 Token slash() |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
222 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
223 switch(source[position]) |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
224 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
225 case '/': |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
226 while(getNextChar != CharType.EOF) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
227 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
228 if(source[position++] == '\n') |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
229 return this.next; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
230 } |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
231 return Token(Tok.EOF, Loc(position), 0); |
1 | 232 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
233 case '*': |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
234 position += 2; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
235 while(getNextChar != CharType.EOF) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
236 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
237 ++position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
238 if(source[position-2] == '*') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
239 if(source[position-1] == '/') |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
240 { |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
241 return this.next; |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
242 } |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
243 } |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
244 messages.report(UnexpectedEOFBlock,Loc(position)); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
245 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
246 case '+': |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
247 position += 2; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
248 int nesting = 1; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
249 while(getNextChar != CharType.EOF) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
250 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
251 ++position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
252 if(source[position-2] == '+') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
253 if(source[position-1] == '/') |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
254 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
255 position++; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
256 nesting--; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
257 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
258 |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
259 if(source[position-2] == '/') |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
260 if(source[position-1] == '+') |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
261 { |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
262 nesting++; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
263 position++; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
264 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
265 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
266 if(nesting == 0) |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
267 return this.next; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
268 } |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
269 messages.report(UnexpectedEOFBlock,Loc(position)); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
270 |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
271 default: |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
272 return Token(Tok.Slash, Loc(position - 1), 1); |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
273 } |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
274 } |
62
78a6808b2e0f
Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents:
43
diff
changeset
|
275 |
74
192da4976daa
Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents:
72
diff
changeset
|
276 Token percent() |
62
78a6808b2e0f
Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents:
43
diff
changeset
|
277 { |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
278 return Token(Tok.Percent, Loc(position - 1), 1); |
62
78a6808b2e0f
Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents:
43
diff
changeset
|
279 } |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
280 |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
281 Token string() |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
282 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
283 --position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
284 int start = position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
285 if(getNextChar() == CharType.Letter) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
286 position++; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
287 char end = '`'; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
288 switch(source[position]) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
289 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
290 case '"': |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
291 if(position > 0) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
292 if(source[position-1] == 'r') |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
293 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
294 end = '"'; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
295 goto string_wys; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
296 } |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
297 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
298 while(getNextChar != CharType.EOF) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
299 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
300 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
301 if (source[position-1] == '"' ) |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
302 { |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
303 if(getNextChar != CharType.EOF) |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
304 if (source[position] == 'c' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
305 source[position] == 'w' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
306 source[position] == 'd') |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
307 position++; |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
308 |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
309 return Token(Tok.String, Loc(start), position - start); |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
310 } |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
311 else if (source[position-1] == '\\') |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
312 position++; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
313 } |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
314 break; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
315 case '`': |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
316 string_wys: |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
317 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
318 while(getNextChar != CharType.EOF) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
319 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
320 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
321 if (source[position-1] == end ) |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
322 return Token(Tok.String, Loc(start), position - start); |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
323 } |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
324 break; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
325 } |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
326 messages.report(UnexpectedEOFBlock, Loc(position)).fatal(ExitLevel.Lexer); |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
327 } |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
328 |
1 | 329 Token lexNumber () |
330 { | |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
331 bool sign; |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
332 |
1 | 333 int i = 0; |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
334 |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
335 |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
336 bool end = false; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
337 while(!end) |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
338 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
339 switch(getNextChar(i)) |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
340 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
341 case CharType.Number: |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
342 break; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
343 case CharType.Symbol: |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
344 if(this.source[position+i] == '.') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
345 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
346 break; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
347 } |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
348 if (this.source[position+i] == '+' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
349 this.source[position+i] == '-') |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
350 { |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
351 if (source[position+i-1] == 'e' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
352 source[position+i-1] == 'E') |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
353 break; |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
354 } |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
355 end = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
356 continue; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
357 case CharType.Letter: |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
358 if(this.source[position+i] == '_') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
359 break; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
360 if (this.source[position+i] == 'e' || |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
361 this.source[position+i] == 'E') |
67
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
362 { |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
363 break; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
364 } |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
365 end = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
366 continue; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
367 |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
368 default: |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
369 end = true; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
370 continue; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
371 } |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
372 i++; |
3fdf20b08a81
Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents:
62
diff
changeset
|
373 } |
1 | 374 |
111
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
375 while(source[position+i] == 'u' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
376 source[position+i] == 'U' || |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
377 source[position+i] == 'L') |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
378 i += 1; |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
379 |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
380 |
c658172ca8a0
Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents:
104
diff
changeset
|
381 |
1 | 382 position += i; |
383 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
384 return Token(Tok.Integer, Loc(position - i), i); |
1 | 385 } |
386 | |
387 Token lexSymbol () | |
388 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
389 Token t = symbolFunctions[source[position++]](); |
1 | 390 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
391 return t; |
1 | 392 } |
393 | |
394 Token lexLetter () | |
395 { | |
396 int i = 0; | |
397 bool hasNumber = false; | |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
398 if (source[position+1] == '"' || |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
399 source[position+1] == '`') |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
400 { |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
401 ++position; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
402 return string; |
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
403 } |
1 | 404 while (getNextChar(++i) == CharType.Letter || |
405 getNextChar(i) == CharType.Number) | |
406 { | |
407 if (getNextChar(i) == CharType.Number) | |
408 { | |
409 hasNumber = true; | |
410 } | |
411 } | |
412 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
413 Token t = Token(Tok.Identifier, Loc(), i); |
1 | 414 |
415 if (!hasNumber) | |
416 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
417 char[] str = source[position .. position + i]; |
1 | 418 if(str in keywords) |
419 t.type = keywords[str]; | |
420 } | |
421 | |
422 position += i; | |
423 | |
424 return t; | |
425 } | |
426 | |
427 CharType getNextChar(int offset = 0) | |
428 { | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
429 if (position + offset >= this.source.length) |
1 | 430 return CharType.EOF; |
431 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
432 char current = source[position + offset]; |
1 | 433 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
434 CharType c = charTable[current]; |
1 | 435 |
43
a712c530b7cc
A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents:
42
diff
changeset
|
436 if(c == CharType.INVALID) |
92
771ac63898e2
A few better parser errors plus renaming most of the sema classes to match that they do now. Some have changes a lot.
Anders Johnsen <skabet@gmail.com>
parents:
91
diff
changeset
|
437 messages.report(InvalidSymbol, Loc()) |
771ac63898e2
A few better parser errors plus renaming most of the sema classes to match that they do now. Some have changes a lot.
Anders Johnsen <skabet@gmail.com>
parents:
91
diff
changeset
|
438 .arg(Integer.toString(cast(int)current)) |
771ac63898e2
A few better parser errors plus renaming most of the sema classes to match that they do now. Some have changes a lot.
Anders Johnsen <skabet@gmail.com>
parents:
91
diff
changeset
|
439 .fatal(ExitLevel.Lexer); |
1 | 440 |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
441 return c; |
1 | 442 |
443 } | |
444 | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
445 private final SourceLocation Loc(int pos = -1) |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
446 { |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
447 if (pos < 0) |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
448 return start_loc + position; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
449 return start_loc + pos; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
450 } |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
451 |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
452 SourceManager sm; |
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
453 SourceLocation start_loc; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
454 int position; |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
455 char[] source; |
89
a49bb982a7b0
Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents:
88
diff
changeset
|
456 MessageHandler messages; |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
457 CharType[] charTable; |
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
458 Token delegate()[] symbolFunctions; |
1 | 459 } |
460 | |
461 enum CharType : ubyte | |
462 { | |
41
f977aa28eb32
Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents:
36
diff
changeset
|
463 INVALID, |
1 | 464 Letter, |
465 Number, | |
466 Symbol, | |
467 Whitespace, | |
104
7ff4bc2accf2
Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents:
99
diff
changeset
|
468 Other, |
1 | 469 |
470 EOF | |
471 } | |
88
eb5b2c719a39
Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents:
74
diff
changeset
|
472 |