Mercurial > projects > dang

annotate lexer/Lexer.d @ 190:85e492318bb6

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Can parse empty file again.
author Anders Johnsen <skabet@gmail.com>
date Fri, 25 Jul 2008 13:59:21 +0200
parents 75d0544ddc45
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
1 module lexer.Lexer;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
2
89
a49bb982a7b0 Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents: 88
diff changeset
3 import basic.Message,
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
4 basic.SourceManager;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
5
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
6 import lexer.Token,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
7 lexer.Keyword;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
8
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
9 import tango.io.Stdout;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
10
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
11 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
12 The Lexer class will supply you with methods to tokenize a D file. Supply the
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
13 Lexer with a DataSource and you can 'peek' and 'next' Tokens from the file.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
14
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
15 For more info about Tokens, look up the lexer.Token module.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
16 */
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
17 class Lexer
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
18 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
19 public:
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
20
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
21 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
22 Create a new Lexer.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
23 */
89
a49bb982a7b0 Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents: 88
diff changeset
24 this(SourceLocation start, SourceManager src_mgr, MessageHandler messages)
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
25 {
89
a49bb982a7b0 Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents: 88
diff changeset
26 this.messages = messages;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
27 sm = src_mgr;
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
28 start_loc = start;
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
29 position = 0;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
30 source = sm.getRawData(start_loc);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
31
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
32
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
33 charTable.length = 256;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
34 foreach (c; "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_")
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
35 charTable[c] = CharType.Letter;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
36
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
37 foreach (c; "0123456789")
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
38 charTable[c] = CharType.Number;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
39
176
dc9bf56b7ace Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents: 154
diff changeset
40 foreach (c; "(){}[];:.,=!<>+-*/%&\"`")
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
41 charTable[c] = CharType.Symbol;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
42
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
43 foreach (c; " \n")
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
44 charTable[c] = CharType.Whitespace;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
45
104
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
46 foreach (c; "'\\")
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
47 charTable[c] = CharType.Other;
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
48
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
49 symbolFunctions.length = 256;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
50
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
51 symbolFunctions['('] = &openParentheses;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
52 symbolFunctions[')'] = &closeParentheses;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
53 symbolFunctions['{'] = &openBrace;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
54 symbolFunctions['}'] = &closeBrace;
72
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
55 symbolFunctions['['] = &openBracket;
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
56 symbolFunctions[']'] = &closeBracket;
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
57 symbolFunctions[';'] = &seperator;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
58 symbolFunctions[':'] = &colon;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
59 symbolFunctions['.'] = &dot;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
60 symbolFunctions[','] = &comma;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
61 symbolFunctions['='] = &eq;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
62 symbolFunctions['!'] = &ne;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
63 symbolFunctions['<'] = &le;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
64 symbolFunctions['>'] = &ge;
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
65 symbolFunctions['+'] = &plus;
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
66 symbolFunctions['-'] = &minus;
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
67 symbolFunctions['*'] = &star;
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
68 symbolFunctions['/'] = &slash;
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
69 symbolFunctions['%'] = &percent;
176
dc9bf56b7ace Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents: 154
diff changeset
70 symbolFunctions['&'] = &and;
104
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
71 symbolFunctions['"'] = &string;
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
72 symbolFunctions['`'] = &string;
190
85e492318bb6 Can parse empty file again.
Anders Johnsen <skabet@gmail.com>
parents: 189
diff changeset
73
85e492318bb6 Can parse empty file again.
Anders Johnsen <skabet@gmail.com>
parents: 189
diff changeset
74 last = Token(Tok.EOF, SLoc() + 1, 0);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
75 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
76
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
77 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
78 Get the next token from the source. This method will move the
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
79 internal position forward to the next Token.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
80
43
a712c530b7cc A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents: 42
diff changeset
81 return: A Token - Token.type is TokType.EOF if there is
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
82 no more tokens in the file.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
83 */
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
84 Token next()
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
85 {
189
75d0544ddc45 Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents: 186
diff changeset
86 Token res;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
87 switch (getNextChar)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
88 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
89 case CharType.EOF:
189
75d0544ddc45 Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents: 186
diff changeset
90 return Token(Tok.EOF, last.location, 0);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
91
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
92 case CharType.Whitespace:
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
93 position += 1;
189
75d0544ddc45 Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents: 186
diff changeset
94 res = this.next;
75d0544ddc45 Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents: 186
diff changeset
95 break;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
96
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
97 case CharType.Symbol:
189
75d0544ddc45 Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents: 186
diff changeset
98 res = lexSymbol;
75d0544ddc45 Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents: 186
diff changeset
99 break;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
100
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
101 case CharType.Letter:
189
75d0544ddc45 Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents: 186
diff changeset
102 res = lexLetter;
75d0544ddc45 Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents: 186
diff changeset
103 break;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
104
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
105 case CharType.Number:
189
75d0544ddc45 Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents: 186
diff changeset
106 res = lexNumber;
75d0544ddc45 Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents: 186
diff changeset
107 break;
104
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
108 case CharType.Other:
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
109 messages.report(UnexpectedTok, Loc(position)).fatal(ExitLevel.Lexer);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
110 }
189
75d0544ddc45 Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents: 186
diff changeset
111 if (res.type != Tok.EOF)
75d0544ddc45 Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents: 186
diff changeset
112 last = res;
75d0544ddc45 Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents: 186
diff changeset
113 return res;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
114 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
115
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
116 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
117 Get the next token from the source. This method will NOT move the
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
118 internal position forward, and thereby having no side-effects.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
119
43
a712c530b7cc A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents: 42
diff changeset
120 return: A Token - Token.type is TokType.EOF if there is
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
121 no more tokens in the file.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
122 */
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
123 Token peek(int skip = 0)
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
124 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
125 int oldPosition = this.position;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
126 while (skip-- > 0)
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
127 this.next;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
128 Token t = this.next;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
129 this.position = oldPosition;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
130 return t;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
131 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
132
189
75d0544ddc45 Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents: 186
diff changeset
133 Token last;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
134 private:
189
75d0544ddc45 Better error handling on unexpected EOF.
Anders Johnsen <skabet@gmail.com>
parents: 186
diff changeset
135
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
136 Token eq()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
137 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
138 if(source[position] == '=')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
139 return Token(Tok.Eq, Loc(position++ - 1), 2);
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
140 return Token(Tok.Assign, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
141 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
142 Token openBrace()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
143 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
144 return Token(Tok.OpenBrace, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
145 }
72
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
146 Token closeBrace()
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
147 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
148 return Token(Tok.CloseBrace, Loc(position - 1), 1);
72
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
149 }
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
150 Token openParentheses()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
151 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
152 return Token(Tok.OpenParentheses, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
153 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
154 Token closeParentheses()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
155 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
156 return Token(Tok.CloseParentheses, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
157 }
72
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
158 Token openBracket()
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
159 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
160 return Token(Tok.OpenBracket, Loc(position - 1), 1);
72
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
161 }
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
162 Token closeBracket()
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
163 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
164 return Token(Tok.CloseBracket, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
165 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
166 Token seperator()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
167 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
168 return Token(Tok.Seperator, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
169 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
170 Token colon()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
171 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
172 return Token(Tok.Colon, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
173 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
174 Token dot()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
175 {
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
176 int pos = 0;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
177 while(getNextChar(0) == CharType.Number ||
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
178 this.source[position + pos + 1] == '_')
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
179 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
180 if(getNextChar(0) == CharType.Number)
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
181 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
182 position--;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
183 return lexNumber();
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
184 }
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
185 pos++;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
186 }
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
187 return Token(Tok.Dot, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
188 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
189 Token comma()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
190 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
191 return Token(Tok.Comma, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
192 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
193 Token ne()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
194 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
195 if(source[position] == '=')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
196 return Token(Tok.Ne, Loc(position++ - 1), 2);
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
197 return Token(Tok.Not, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
198 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
199 Token le()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
200 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
201 if(source[position] == '=')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
202 return Token(Tok.Le, Loc(position++ - 1), 2);
123
6a5f745d351c Parsing <<, >> and >>>.
Anders Johnsen <skabet@gmail.com>
parents: 111
diff changeset
203 if(source[position] == '<')
6a5f745d351c Parsing <<, >> and >>>.
Anders Johnsen <skabet@gmail.com>
parents: 111
diff changeset
204 return Token(Tok.LeftShift, Loc(position++ - 1), 2);
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
205 return Token(Tok.Lt, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
206 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
207 Token ge()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
208 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
209 if(source[position] == '=')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
210 return Token(Tok.Ge, Loc(position++ - 1), 2);
123
6a5f745d351c Parsing <<, >> and >>>.
Anders Johnsen <skabet@gmail.com>
parents: 111
diff changeset
211 if(source[position] == '>')
6a5f745d351c Parsing <<, >> and >>>.
Anders Johnsen <skabet@gmail.com>
parents: 111
diff changeset
212 if(source[position+1] == '>')
6a5f745d351c Parsing <<, >> and >>>.
Anders Johnsen <skabet@gmail.com>
parents: 111
diff changeset
213 {
6a5f745d351c Parsing <<, >> and >>>.
Anders Johnsen <skabet@gmail.com>
parents: 111
diff changeset
214 position += 2;
6a5f745d351c Parsing <<, >> and >>>.
Anders Johnsen <skabet@gmail.com>
parents: 111
diff changeset
215 return Token(Tok.UnsignedRightShift, Loc(position - 1), 3);
6a5f745d351c Parsing <<, >> and >>>.
Anders Johnsen <skabet@gmail.com>
parents: 111
diff changeset
216 }
6a5f745d351c Parsing <<, >> and >>>.
Anders Johnsen <skabet@gmail.com>
parents: 111
diff changeset
217 else
6a5f745d351c Parsing <<, >> and >>>.
Anders Johnsen <skabet@gmail.com>
parents: 111
diff changeset
218 return Token(Tok.RightShift, Loc(position++ - 1), 2);
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
219 return Token(Tok.Gt, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
220 }
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
221 Token plus()
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
222 {
126
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents: 123
diff changeset
223 if(source[position] == '=')
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents: 123
diff changeset
224 return Token(Tok.PlusAssign, Loc(position++ - 1), 2);
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
225 return Token(Tok.Plus, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
226 }
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
227 Token minus()
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
228 {
126
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents: 123
diff changeset
229 if(source[position] == '=')
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents: 123
diff changeset
230 return Token(Tok.MinusAssign, Loc(position++ - 1), 2);
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
231 return Token(Tok.Minus, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
232 }
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
233 Token star()
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
234 {
126
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents: 123
diff changeset
235 if(source[position] == '=')
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents: 123
diff changeset
236 return Token(Tok.StarAssign, Loc(position++ - 1), 2);
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
237 return Token(Tok.Star, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
238 }
104
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
239 Token slash()
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
240 {
186
e1e170c2cd44 Fixed a error in the test program.
Anders Johnsen <skabet@gmail.com>
parents: 176
diff changeset
241 int p = position;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
242 switch(source[position])
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
243 {
126
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents: 123
diff changeset
244 case '=':
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents: 123
diff changeset
245 return Token(Tok.SlashAssign, Loc(position++ - 1), 2);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
246 case '/':
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
247 while(getNextChar != CharType.EOF)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
248 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
249 if(source[position++] == '\n')
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
250 return this.next;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
251 }
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
252 return Token(Tok.EOF, Loc(position), 0);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
253
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
254 case '*':
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
255 position += 2;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
256 while(getNextChar != CharType.EOF)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
257 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
258 ++position;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
259 if(source[position-2] == '*')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
260 if(source[position-1] == '/')
104
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
261 {
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
262 return this.next;
104
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
263 }
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
264 }
186
e1e170c2cd44 Fixed a error in the test program.
Anders Johnsen <skabet@gmail.com>
parents: 176
diff changeset
265 messages.report(UnexpectedEOFBlock,Loc(p)).fatal(ExitLevel.Lexer);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
266
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
267 case '+':
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
268 position += 2;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
269 int nesting = 1;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
270 while(getNextChar != CharType.EOF)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
271 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
272 ++position;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
273 if(source[position-2] == '+')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
274 if(source[position-1] == '/')
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
275 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
276 position++;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
277 nesting--;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
278 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
279
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
280 if(source[position-2] == '/')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
281 if(source[position-1] == '+')
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
282 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
283 nesting++;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
284 position++;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
285 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
286
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
287 if(nesting == 0)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
288 return this.next;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
289 }
136
2be29b296081 Lots of changes:
johnsen@johnsen-laptop
parents: 126
diff changeset
290 messages.report(
2be29b296081 Lots of changes:
johnsen@johnsen-laptop
parents: 126
diff changeset
291 UnexpectedEOFBlock,
186
e1e170c2cd44 Fixed a error in the test program.
Anders Johnsen <skabet@gmail.com>
parents: 176
diff changeset
292 Loc(p)).fatal(ExitLevel.Lexer);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
293
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
294 default:
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
295 return Token(Tok.Slash, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
296 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
297 }
176
dc9bf56b7ace Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents: 154
diff changeset
298 Token and()
dc9bf56b7ace Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents: 154
diff changeset
299 {
dc9bf56b7ace Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents: 154
diff changeset
300 return Token(Tok.And, Loc(position - 1), 1);
dc9bf56b7ace Can now use & as a unary operator and take an AddressOf
Anders Johnsen <skabet@gmail.com>
parents: 154
diff changeset
301 }
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
302 Token percent()
62
78a6808b2e0f Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents: 43
diff changeset
303 {
126
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents: 123
diff changeset
304 if(source[position] == '=')
c3b24e7e8cf8 Carius changes to the parser. Parsing attributes, lexing many keywords(not all yet).
Anders Johnsen <skabet@gmail.com>
parents: 123
diff changeset
305 return Token(Tok.PercentAssign, Loc(position++ - 1), 2);
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
306 return Token(Tok.Percent, Loc(position - 1), 1);
62
78a6808b2e0f Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents: 43
diff changeset
307 }
104
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
308
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
309 Token string()
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
310 {
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
311 --position;
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
312 int start = position;
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
313 if(getNextChar() == CharType.Letter)
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
314 position++;
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
315 char end = '`';
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
316 switch(source[position])
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
317 {
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
318 case '"':
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
319 if(position > 0)
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
320 if(source[position-1] == 'r')
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
321 {
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
322 end = '"';
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
323 goto string_wys;
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
324 }
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
325 ++position;
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
326 while(getNextChar != CharType.EOF)
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
327 {
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
328 ++position;
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
329 if (source[position-1] == '"' )
111
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
330 {
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
331 if(getNextChar != CharType.EOF)
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
332 if (source[position] == 'c' ||
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
333 source[position] == 'w' ||
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
334 source[position] == 'd')
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
335 position++;
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
336
104
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
337 return Token(Tok.String, Loc(start), position - start);
111
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
338 }
104
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
339 else if (source[position-1] == '\\')
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
340 position++;
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
341 }
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
342 break;
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
343 case '`':
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
344 string_wys:
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
345 ++position;
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
346 while(getNextChar != CharType.EOF)
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
347 {
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
348 ++position;
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
349 if (source[position-1] == end )
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
350 return Token(Tok.String, Loc(start), position - start);
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
351 }
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
352 break;
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
353 }
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
354 messages.report(UnexpectedEOFBlock, Loc(position)).fatal(ExitLevel.Lexer);
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
355 }
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
356
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
357 Token lexNumber ()
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
358 {
111
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
359 bool sign;
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
360
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
361 int i = 0;
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
362
111
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
363
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
364 bool end = false;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
365 while(!end)
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
366 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
367 switch(getNextChar(i))
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
368 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
369 case CharType.Number:
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
370 break;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
371 case CharType.Symbol:
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
372 if(this.source[position+i] == '.')
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
373 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
374 break;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
375 }
111
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
376 if (this.source[position+i] == '+' ||
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
377 this.source[position+i] == '-')
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
378 {
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
379 if (source[position+i-1] == 'e' ||
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
380 source[position+i-1] == 'E')
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
381 break;
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
382 }
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
383 end = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
384 continue;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
385 case CharType.Letter:
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
386 if(this.source[position+i] == '_')
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
387 break;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
388 if (this.source[position+i] == 'e' ||
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
389 this.source[position+i] == 'E')
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
390 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
391 break;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
392 }
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
393 end = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
394 continue;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
395
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
396 default:
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
397 end = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
398 continue;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
399 }
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
400 i++;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
401 }
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
402
111
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
403 while(source[position+i] == 'u' ||
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
404 source[position+i] == 'U' ||
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
405 source[position+i] == 'L')
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
406 i += 1;
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
407
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
408
c658172ca8a0 Parsing basic integers and floats.
Anders Johnsen <skabet@gmail.com>
parents: 104
diff changeset
409
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
410 position += i;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
411
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
412 return Token(Tok.Integer, Loc(position - i), i);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
413 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
414
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
415 Token lexSymbol ()
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
416 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
417 Token t = symbolFunctions[source[position++]]();
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
418
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
419 return t;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
420 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
421
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
422 Token lexLetter ()
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
423 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
424 int i = 0;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
425 bool hasNumber = false;
104
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
426 if (source[position+1] == '"' ||
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
427 source[position+1] == '`')
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
428 {
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
429 ++position;
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
430 return string;
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
431 }
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
432 while (getNextChar(++i) == CharType.Letter ||
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
433 getNextChar(i) == CharType.Number)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
434 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
435 if (getNextChar(i) == CharType.Number)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
436 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
437 hasNumber = true;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
438 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
439 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
440
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
441 Token t = Token(Tok.Identifier, Loc(), i);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
442
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
443 if (!hasNumber)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
444 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
445 char[] str = source[position .. position + i];
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
446 if(str in keywords)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
447 t.type = keywords[str];
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
448 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
449
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
450 position += i;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
451
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
452 return t;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
453 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
454
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
455 CharType getNextChar(int offset = 0)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
456 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
457 if (position + offset >= this.source.length)
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
458 return CharType.EOF;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
459
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
460 char current = source[position + offset];
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
461
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
462 CharType c = charTable[current];
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
463
43
a712c530b7cc A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents: 42
diff changeset
464 if(c == CharType.INVALID)
92
771ac63898e2 A few better parser errors plus renaming most of the sema classes to match that they do now. Some have changes a lot.
Anders Johnsen <skabet@gmail.com>
parents: 91
diff changeset
465 messages.report(InvalidSymbol, Loc())
771ac63898e2 A few better parser errors plus renaming most of the sema classes to match that they do now. Some have changes a lot.
Anders Johnsen <skabet@gmail.com>
parents: 91
diff changeset
466 .arg(Integer.toString(cast(int)current))
771ac63898e2 A few better parser errors plus renaming most of the sema classes to match that they do now. Some have changes a lot.
Anders Johnsen <skabet@gmail.com>
parents: 91
diff changeset
467 .fatal(ExitLevel.Lexer);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
468
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
469 return c;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
470
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
471 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
472
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
473 private final SourceLocation Loc(int pos = -1)
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
474 {
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
475 if (pos < 0)
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
476 return start_loc + position;
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
477 return start_loc + pos;
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
478 }
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
479
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
480 SourceManager sm;
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
481 SourceLocation start_loc;
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
482 int position;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
483 char[] source;
89
a49bb982a7b0 Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents: 88
diff changeset
484 MessageHandler messages;
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
485 CharType[] charTable;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
486 Token delegate()[] symbolFunctions;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
487 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
488
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
489 enum CharType : ubyte
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
490 {
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
491 INVALID,
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
492 Letter,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
493 Number,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
494 Symbol,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
495 Whitespace,
104
7ff4bc2accf2 Lexing more types of strings. Now all's left is to parse the string in the AST.
Anders Johnsen <skabet@gmail.com>
parents: 99
diff changeset
496 Other,
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
497
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
498 EOF
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
499 }
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
500